Cloud performance

At times during testing of our ACI Windows prototypes, we have experienced some less that ideal performance results. This has included the time to complete tasks such as log-in, application launch, and data upload/download. In and of itself, this is not a concern and may only require we utilize larger server instances with more available CPU and memory. However what was unexpected was the inconsistency of the results. Two identical instances (cpu, memory, storage, OS, patch level, applications, policies, etc) should provide very similar (good or bad) test results.

Webmetrics financed Bitcurrent to research a month long study of the performance of five of the major public cloud providers. They recently published their results.

While it doesn't explain the inconsistent results, it does help to understand the strengths and weaknesses of each vendor's approach and architecture. This kind of information, when considered with cost and management capabilities, will help to determine the right vendor for a given use case. As to the consistency concern, we'll need to begin to chart activity to assess what is the most likely source of the issue to determine the best approach for a solution.

Windows ACI: Stu's weeping

That we will utilize Microsoft's Active Directory (AD) to manage our server instances and users (accounts and profiles) has been the assumption from the beginning of this experiment. How best to accomplish this is a very challenging question. Hence the label: "Stu's weeping".

There are several reasons for using AD as part of our management strategy. First, it facilitates a single sign-on (SSO) solution. Without it, users would first need to log into the TS Gateway and then to their individual Terminal Server instance. Second it allows us to centrally manage user accounts, user profiles, and user and server configurations via group policies. And it permits us to leverage our existing management practices and procedures.

If we moved to production and on average only supported 2-3 TS instances and 5-25 user accounts, well then this would probably be overkill. However, we believe that if successful, we will likely be supporting many TS server instances with potentially hundreds of user accounts.

We've spent the better part of the last several weeks working to figure out whether the better approach is to create an AD structure within the cloud or try and join our cloud based servers to the University's test AD forest, under the ICPSR Organizational Unit (OU). We believe the better route is to join them to our on premise AD but it has been more difficult than anticipated for a variety of reasons which I'll explain in the next post.

Windows ACI: Terminal Server Gateway

I've elected to setup a Terminal Server Gateway as the entry point for access to the ACIs. This server role was introduced with Windows Server 2008 and provides several benefits that will help with the management and security of the ACIs. First and foremost, a TS Gateway provides a means to encapsulate the Remote Desktop Protocol (RDP) traffic through an SSL tunnel using HTTPS over port 443. In turn the TS Gateway facilitates a standard RDP connection, over port 3389, to the requested resource. This allows us to configure the firewall of our cloud service provider account and the individual server instances to isolate the ACIs to only be reachable via the TS Gateway and a few IP addresses here at ICPSR. The TS Gateway also provides for the implementation of client and resource access policies to control which resources can be accessed and by whom, therefore we can restrict access to individual ACIs to strictly the users identified in the associated research request. Additionally, the TS Gateway can be configured to utilize a Network Access Protection (NAP) health policy. This technology allows for us to define client "health" condition in order to be permitted to connect, for example that the client's firewall is enabled, current for operating system security patches, has anti-virus software.

Implementation of this server role should help to simplify support in a production scenario where there could be a large number of ACIs active at any given time.

Windows ACI: Terminal Server

Our 'concept' Windows ACI is using Windows Terminal Services. This technology, renamed Remote Desktop Services with the release of Windows Server 2008 R2, has been available as a service/role of the Windows Server operating systems since NT. Its maturity has allowed for a great deal of documentation to be developed about managing it, the quirks, and its strengths and weaknesses. This provided us a sort of head start to assessing the challenges of leveraging it to delivery the robust environment we have envisioned. We are not using the R2 version, hence we continue to refer to it as Terminal Server, due to the fact that Server 2008 R2 is not supported (at least currently) by the major cloud computing service providers. That is fine as it does not appear that any of the enhancements or features provided with the R2 release offer any additional value to our project.

Terminal Server is Microsoft's solution for server based computing (SBC). I find that ironic given it was Microsoft's popularization of desktop computing that lead to it replacing the original SBC... the mainframe. Using SBC allows us to provision access to requested research data by instead focusing on providing the working environment for the researchers to perform their analysis. Researchers can remotely access a server we create for them. Once logged in, they are provided a standard Windows-style desktop with the tools and applications they require to perform their work, using a copy of the requested data. The remote session or connection to the server will appear as just another 'window' on their own desktop or laptop computer. They will not be able to move items to/from the remote server to their computer, but will be able to interact with all the applications installed on the server as they normally would.

The question of course is can we satisfy our security requirements while giving the researcher a rich environment with good performance?

Windows ACI: Concept or Prototype?

I'm not certain which descriptor fits but regardless we have in place a test version of our Windows ACI. At this point its a pretty basic setup of a Windows 2008 server configured to perform the Terminal Server role. I've installed three of the statistical software packages used here at ICPSR along with a few productivity applications such as Word and Excel. Test data has been placed on the device. We are using a mandatory user profile and researchers will connect to their ACIs via a separate Windows 2008 server configured as a Terminal Server Gateway. Its our intent that the TS Gateway will be the only Internet accessible device but we are still working out all the assorted details.

Felicia is current giving it a test spin and I am developing a questionnaire intended to capture observations from testers of the system. This document will likely evolve as we proceed.

I think concept is probably more accurate as this test ACI is extremely rough around the edges and will most assuredly change a great deal before we begin piloting.

What we've been doing during our spring break

The blog has been quiet, but the team has been busy working on components of the system.

Steve has built a prototype of our ACI Chooser, the web application that enables a researcher to link an executed restricted-use contract with ACI preferences, such as operating system, stat package, etc.

Steve has also built an early version of the ACI Launcher. This tool uses preferences set by the researcher, and builds an AWS instance to support the research. The tool also enforces ICPSR's license and access controls related to the research data.

Unlike the world of Linux where it has been pretty easy to define and launch a locked-down, managed AWS instance, the world of Windows has been more difficult. Since joining ICPSR earlier this year, Stu has made great progress building an early version of a Windows-flavored ACI to go along with the Linux-flavored one Steve has already built. Stu has also been researching the best way to tie our ACIs with necessary Windows infrastructure, like Active Directory. We often see Stu weeping, and we are not sure if it is with joy or frustration.

The Windows ACI: Decisions...

There are a number of technologies available for developing and managing virtual Windows computing environments. We decided to focus initially on products available from Microsoft and will then compare against the benefits of alternatives such as Citrix and VMWare.

The design we are considering would leverage the University of Michigan Active Directory (AD) infrastructure for user and computer account management. Residing in the cloud would be a public facing Remote Desktop (RD) Gateway through which users of this system would access their research group's ACI. Each ACI would be a Remote Desktop Services (RDS) server, formerly known as Terminal Server, configured based on the information provided by their PI through the ACI Chooser. The RD Gateway would validate the users credentials and redirect them to their assigned ACI.

This approach allows us to managed a single entry point for access to the cloud based resources and utilizing AD permits policy based management of the configuration of the Windows ACIs and associated user accounts. However a requirement of this approach is that a connection must be permitted to the UM network for the RD Gateway residing in the cloud. In the short term we are working with University engineers to permit this connection. Long term we will evaluate alternative approaches to facilitate this design, such as: using dedicated VPN appliance gateways to establish the connection; foregoing the connection to the UM AD and instead create an AD infrastructure in the cloud; and determine whether we should setup a Virtual Private Cloud (VPC) in the EC2 as an added layer for security. Right now our focus is on developing a working prototype RDS that can be tested by our PI and the ICPSR staff.

My next few blog posts will drill down in the merits of this design, its components and their roles, and the decisions required if we were to proceed with a production implementation of this approach.

Customizing an ACI

One of the tasks associated with bring up an ACI is configuring it for a particular group of researchers. There are (at least) three steps that have to be taken:

• create user accounts
• make required statistical software available
• copy the specified dataset(s) to the ACI

My plan was to create an archive containing the following pieces:

• a list of users for whom accounts needed to be created
• a list of statistical packages to be activated
• an inner archive containing the dataset(s)
• a script which would use the two lists to create accounts and activate software, and then expand the dataset archive

The ACI creation script would then pass that archive to the instance via the '--user-data-file' option of the 'ec2-run-instances' script. At first run, the ACI will fetch the user data, unpack it, and execute the included script.

Unfortunately, there is a 16,384-byte limit on the size of the user data, which meant that this approach was not practical.

My current idea is to create two archives. The main archive is as described above; the second is a bootstrap archive that will contain:

• a pointer to a location from which the main archive should be fetched
• a set of credentials that can be used to do that fetching
• a script that can use those credentials to fetch the specified archive

The bootstrap archive will be small enough to be passed to the instance via the '--user-data-file' option of the 'ec2-run-instances' script. At first run, the ACI will fetch the bootstrap archive, unpack it, and execute its included script, fetching the main archive. Once the main archive has been fetched, the credentials will be destroyed, the just-fetched archive will be unpacked, and its included script executed.

I'm working on that process now.