New Collaborative Editing for Amazon WorkDocs – Powered by Hancom Thinkfree Office Online

I’ve got some important news for Amazon WorkDocs users. As a result of our partnership with Hancom, you can now edit Microsoft Office documents in your browser without having to install any applications or connect with another web service. You can quickly create a document, share it with team members, and let them make changes and contribute to the finished product. Everyone can see changes in real-time as they work together, regardless of where they are located or what device they are using to access WorkDocs.

This feature is available at no extra charge and you can start using it as soon as your WorkDocs administrator enables it. Let’s take a tour!

Collaborative Editing
I start by creating a document, spreadsheet, or presentation using the New menu. I’ll create a document:

I can create and edit my document from the comfort of my web browser:

Then I save and rename it (a default name is generated using the creation time as a starting point):

Next, I share it with my colleague Manoj so that he can take a look and make any desired edits:

I can see his edits in real-time:

And I can see all of the participants in the collaborative editing session:

WorkDocs creates a new revision after all of the participants have exited the editing session.

I can also create new spreadsheets and presentations and edit existing ones! Here’s a new spreadsheet:

And here’s an existing presentation (I opened one from 2008 just for fun):

Now Available
This feature is available now in the US West (Oregon) Region and will become available in other regions in the next couple of weeks. It is available at no extra charge to all WorkDocs users.

Jeff;

Amazon EC2 Update – Additional Instance Types, Nitro System, and CPU Options

I have a backlog of EC2 updates to share with you. We’ve been releasing new features and instance types at a rapid clip and it is time to catch up. Here’s a quick peek at where we are and where we are going…

Additional Instance Types
Here’s a quick recap of the most recent EC2 instance type announcements:

Compute-Intensive – The compute-intensive C5d instances provide a 25% to 50% performance improvement over the C4 instances. They are available in 5 regions and offer up to 72 vCPUs, 144 GiB of memory, and 1.8 TB of local NVMe storage.

General Purpose – The general purpose M5d instances are also available in 5 regions. They offer up to 96 vCPUs, 384 GiB of memory, and 3.6 TB of local NVMe storage.

Bare Metal – The i3.metal instances became generally available in 5 regions a couple of weeks ago. You can run performance analysis tools that are hardware-dependent, workloads that require direct access to bare-metal infrastructure, applications that need to run in non-virtualized environments for licensing or support reasons, and container environments such as Clear Containers, while you take advantage of AWS features such as Elastic Block Store (EBS), Elastic Load Balancing, and Virtual Private Clouds. Bare metal instances with 6 TB, 9 TB, 12 TB, and more memory are in the works, all designed specifically for SAP HANA and other in-memory workloads.

Innovation and the Nitro System
The Nitro system is a rich collection of building blocks that can be assembled in many different ways, giving us the flexibility to design and rapidly deliver EC2 instance types with an ever-broadening selection of compute, storage, memory, and networking options. We will deliver new instance types more quickly than ever in the months to come, with the goal of helping you to build, migrate, and run even more types of workloads.

Local NVMe Storage – The new C5d, M5d, and bare metal EC2 instances feature our Nitro local NVMe storage building block, which is also used in the Xen-virtualized I3 and F1 instances. This building block provides direct access to high-speed local storage over a PCI interface and transparently encrypts all data using dedicated hardware. It also provides hardware-level isolation between storage devices and EC2 instances so that bare metal instances can benefit from local NVMe storage.

Nitro Security Chip – A component that is part of our AWS server designs that continuously monitors and protects hardware resources and independently verifies firmware each time a system boots.

Nitro Hypervisor – A thin, quiescent hypervisor that manages memory and CPU allocation, and delivers performance that is indistinguishable from bare metal for most workloads (Brendan Gregg of Netflix benchmarked it at less than 1%).

Networking – Hardware support for the software defined network inside of each Virtual Private Cloud (VPC), Enhanced Networking, and Elastic Network Adapter.

Elastic Block Storage – Hardware EBS processing including CPU-intensive cryptographic operations.

Moving storage, networking, and security functions to hardware has important consequences for both bare metal and virtualized instance types:

Virtualized instances can make just about all of the host’s CPU power and memory available to the guest operating systems since the hypervisor plays a greatly diminished role.

Bare metal instances have full access to the hardware, but also have the same the flexibility and feature set as virtualized EC2 instances including CloudWatch metrics, EBS, and VPC.

To learn more about the hardware and software that make up the Nitro system, watch Amazon EC2 Bare Metal Instances or C5 Instances and the Evolution of Amazon EC2 Virtualization and take a look at The Nitro Project: Next-Generation EC2 Infrastructure.

CPU Options
This feature provides you with additional control over your EC2 instances and lets you optimize your instance for a particular workload. First, you can specify the desired number of vCPUs at launch time. This allows you to control the vCPU to memory ratio for Oracle and SQL Server workloads that need high memory, storage, and I/O but perform well with a low vCPU count. As a result, you can optimize your vCPU-based licensing costs when you Bring Your Own License (BYOL). Second, you can disable Intel® Hyper-Threading Technology (Intel® HT Technology) on instances that run compute-intensive workloads. These workloads sometimes exhibit diminished performance when Intel HT is enabled. Both of these options are available when you launch an instance using the AWS Command Line Interface (CLI) or one of the AWS SDKs. You simply specify the total number of cores and the number of threads per core using values chosen from the CPU Cores and Threads per CPU Core Per Instance Type table. Here’s how you would launch an instance with 6 CPU cores and Intel® HT Technology disabled:

$ aws ec2 run-instances --image-id ami-1a2b3c4d --instance-type r4.4xlarge --cpu-options "CoreCount=6,ThreadsPerCore=1"

To learn more, read about Optimizing CPU Options.

Help Wanted
The EC2 team is always hiring! Here are a few of their open positions:

Jeff;

Amazon Polly Plugin for WordPress Update – Translate and Vocalize Your Content

Earlier this year I showed you how to Give Your WordPress Blog a Voice with Amazon Polly and walked you through the steps involved in installing, configuring, and using the Amazon Polly for WordPress plugin. Today we are making this plugin even more powerful, adding the ability to translate your content into one or more languages and to produce audio versions of each translation. The translation is implemented using Amazon Translate, a neural machine translation service that is part of our portfolio of machine learning services.

The original version of the plugin works like this:

And the new version works like this:

This version of the plugin supports translation of English-language web content into Spanish, German, French, and Portuguese, with plans to support other languages in the future.

Updating and Configuring the Plugin
My earlier post covered the steps involved in launching an Amazon Lightsail instance and setting up the plugin, and I won’t repeat them here. The first step is to edit my existing IAM policy so that it allows calls to the TranslateText function:

Then I log in to the WordPress Admin dashboard, click Plugins, and see that a new version is available:

I click update now, and wait a few seconds for the update. Then I click Settings to enable translation:

I click Enable translation support and Save Changes, then come back and set up the details. I select all of the available target languages, leave the voices and labels as-is, and click Save Changes to move forward:

Creating Translations and Vocalizations
Now I can create a new post and exercise the plugin. I enter the title and text for the post as usual:

Before moving forward, I can click How much will this cost to convert? to check on costs.

The price seems reasonable to me. I publish the post, and then click Translate to generate audio in 4 other languages. This happens in a matter of seconds:

The published post now includes a player that lets me listen to the original audio or any of the 4 translations:

Here are the audio versions:

English:
Spanish:
German:
French:
Portuguese:

I have lots of customization options. For example, I can enable transcripts of the translated text:

The transcripts are shown in the post:

I can change the labels that are used for each language:

Here are the updated labels:

I can also specify the Polly voice for each target language:

Now Available
The updated plugin is available now and you can start using it today! As you can see, it uses the “magic” of machine translation and text-to-speech to make your web content accessible to a wider audience, in both written and spoken form.

Jeff;

 

New – Redis 4.0 Compatibility in Amazon ElastiCache

Amazon ElastiCache makes it easy for you to set up a fully managed in-memory data store and cache with Redis or Memcached. Today we’re pleased to launch compatibility with Redis 4.0 in ElastiCache. You can now launch Redis 4.0 compatible ElastiCache nodes or clusters, in all commercial AWS regions. ElastiCache Redis clusters can scale to terabytes of memory and millions of reads / writes per second to serve the most demanding needs of games, IoT devices, financial applications, and web applications.

Launching a Redis cluster in the AWS Management Console or AWS Command Line Interface (CLI) remains simple. I’m going to create a small cluster to play with the new Redis 4.0 features, to use the new version I just select a 4.0 release in “Engine version compatibility”. This will launch, at the time of this writing, a 4.0.10 compatible cluster.

New Features

  • Least Frequently Used (LFU) cache eviction policy – Redis 4.0 launched with a number of caching improvements including a new LFU cache eviction algorithm, customers may see better performance from LFU over Least Recently Used (LRU). Antirez’s blog has a deep dive on some of the changes.
  • Asynchronous FLUSHDB, FLUSHALL, and UNLINK – using the ASYNC option of the FLUSH commands allows users to make a non-blocking call to clear databases. Using UNLINK instead of DEL allows users to asynchronously delete individual keys. There’s also the SWAPDB command which can be useful to atomically switch between entire datasets.
  • Active memory defragmentation – Redis can now defragment memory while running which allows more efficient utilization of memory for customer data. This is off by default but you can modify the parameter group to turn it on. Customers should probably only turn it on if they’re running into fragmentation issues.
  • Online Cluster Resizing and Encryption in transit – with Redis 4.0 you can now use encryption in transit and online cluster resizing at the same time. With Online Cluster Resizing you can add and remove shards from a running cluster to dynamically scale-out or scale-in your Redis cluster and adapt to changes on demand. Previously this feature wasn’t able to be used with encryption in transit but now you can use both features simultaneously. This helps with workloads that require encryption for compliance purposes.
  • MEMORY commands – a whole new family of memory commands: DOCTOR, USAGE, STATS, PURGE, and MALLOC-STATS are available for gathering statistics or usage information on your redis nodes. Running MEMORY DOCTOR will tell you about any memory issues (and it will give you a nice sci-fi easter egg if no problems are detected). The MEMORY STATS command will return some useful statistics like “bytes-per-key” that aren’t available in the INFO commands.

Additional Resources

You can find more information in the documentation and in antirez’s blogs/release notes.

We hope customers can take advantage of these new features right away. As always, feel free to leave any comments below or reach out to us on twitter!

Randall

AWS DeepLens Now Shipping – Order One Today!

AWS DeepLens is a video camera that runs deep learning models directly on the device, out in the field. I wrote about the hardware and system software in depth last year; here’s a quick recap:

Hardware – 4 megapixel camera (1080P video), 2D microphone array, Intel Atom® Processor, dual-band Wi-Fi, USB and micro HDMI ports, 8 GB of memory for models and code.

Software – Ubuntu 16.04, AWS Greengrass Core, device-optimized versions of MXNet and Intel® clDNN library, support for other deep learning frameworks.

The response to this AWS re:Invent was immediate and gratifying! Educators, students, and developers signed up for hands-on sessions and started to build and train models right away. Their enthusiasm continued throughout the preview period and into this year’s AWS Summit season, where we did our best to provide all interested parties with access to devices, tools, and training.

Hackathons and Challenges
We made DeepLens devices available to participants in last month’s HackTillDawn. I was fortunate enough to be able to attend the event and to help to choose the three winners. It was amazing to watch the teams, most with no previous machine learning or computer vision experience, dive right in and build interesting, sophisticated applications designed to enhance the attendee experience at large-scale music festivals. The three winners went on to compete at EDC Vegas, where the Grand Prize winner (Find Your Totem) was chosen. Congrats to the team, and have fun at EDC Orlando!

We also ran the AWS DeepLens Challenge, asking participants to build machine learning projects that made use of DeepLens, with bonus points for the use of Amazon SageMaker and/or AWS Lambda. The submissions were as diverse as they were interesting, with applications designed for children, adults, and animals. Details on all of the submissions, including demo videos and source code, are available on the Community Projects page. The three winning applications were ReadToMe (first place), Dee (second place), and SafeHaven (third place).

From what I can tell, DeepLens has proven itself as an excellent learning vehicle. While speaking to the attendees at HackTillDawn, I learned that many of them were eager to get some hands-on experience that they could use to broaden their skillsets and to help them to progress in their careers.

Preview Updates
During the preview period, the DeepLens team has stayed heads-down, focusing on making the device even more capable. Significant additions include:

Gluon Support – Computer vision models can be built using Gluon (an imperative interface to MXNet), trained, imported to DeepLens, and deployed.

SageMaker Import – Models can be built and trained in Amazon SageMaker and then imported to DeepLens.

Model Optimizer – The optimizer runs on the device and optimizes downloaded MXNet models so that they run efficiently on the DeepLens GPU.

Now Shipping
I am happy to report that DeepLens is now shipping and available to order from Amazon.com. You can get one of your very own and start building your own deep learning applications within days. Devices can be shipped to addresses in the United States, with additional destinations in the works.

We are also rounding out the initial feature set with the addition of some important new capabilities:

Expanded Framework Support – DeepLens now supports the TensorFlow and Caffe frameworks.

Expanded MXNet Layer Support – DeepLens now supports the Deconvolution, L2Normalization, and LRN layers provided by MXNet.

Kinesis Video Streams – The video stream from the DeepLens camera can now be used in conjunction with Amazon Kinesis Video Streams. You can stream the raw camera feed to the cloud and then use Amazon Rekognition Video to extract objects, faces, and content from the video.

New Sample Project – DeepLens now includes a sample project for head pose detection (powered by TensorFlow). You can examine this sample to see how the model was constructed; here’s an excerpt from the notebook:

I am looking forward to seeing what you build with your very own DeepLens. Drop me a line and let me know!

Jeff;

Amazon SageMaker Automatic Model Tuning: Using Machine Learning for Machine Learning

Today I’m excited to announce the general availability of Amazon SageMaker Automatic Model Tuning. Automatic Model Tuning eliminates the undifferentiated heavy lifting required to search the hyperparameter space for more accurate models. This feature allows developers and data scientists to save significant time and effort in training and tuning their machine learning models. A Hyperparameter Tuning job launches multiple training jobs, with different hyperparameter combinations, based on the results of completed training jobs. SageMaker trains a “meta” machine learning model, based on Bayesian Optimization, to infer hyperparameter combinations for our training jobs. Let’s dive a little deeper.

Model Tuning in the Machine Learning Process

A developer’s typical machine learning process comprises 4 steps: exploratory data analysis (EDA), model design, model training, and model evaluation. SageMaker already makes each of those steps easy with access to powerful Jupyter notebook instances, built-in algorithms, and model training within the service. Focusing on the training portion of the process, we typically work with data and feed it into a model where we evaluate the model’s prediction against our expected result. We keep a portion of our overall input data, the evaluation data, away from the training data used to train the model. We can use the evaluation data to examine the behavior of our model on data it has never seen. In many cases after we’ve chosen an algorithm or built a custom model, we will need to search the space of possible hyperparameter configurations of that algorithm for the best results for our input data.

Hyperparameters control how our underlying algorithms operate and influence the performance of the model. They can be things like: the number of epochs to train for, the number of layers in the network, the learning rate, the optimization algorithms, and more. Typically, you start with random values, or common values for other problems, and iterate through adjustments as you begin to see what effect the changes have. In the past this was a painstakingly manual process. However, thanks to the work of some very talented researchers we can use SageMaker to eliminate almost all of the manual overhead. A user only needs to select the hyperparameters to tune, a range for each parameter to explore, and the total number of training jobs to budget. Let’s see how this works in practice.

Hyperparameter Tuning

To demonstrate this feature we’ll work with the standard MNIST dataset, the Apache MXNet framework, and the SageMaker Python SDK. Everything you see below is available in the SageMaker example notebooks.

First, I’ll create a traditional MXNet estimator using the SageMaker Python SDK on a Notebook Instance:


import boto3
import sagemaker
from sagemaker.mxnet import MXNet
role = sagemaker.get_execution_role()
region = boto3.Session().region_name
train_data_location = 's3://sagemaker-sample-data-{}/mxnet/mnist/train'.format(region)
test_data_location = 's3://sagemaker-sample-data-{}/mxnet/mnist/test'.format(region)
estimator = MXNet(entry_point='mnist.py',
                  role=role,
                  train_instance_count=1,
                  train_instance_type='ml.m4.xlarge',
                  sagemaker_session=sagemaker.Session(),
                  base_job_name='HPO-mxnet',
                  hyperparameters={'batch_size': 100})

This is probably quite similar to what you’ve seen in other SageMaker examples.

Now, we can import some tools for the Auto Model Tuning and create our hyperparameter ranges.


from sagemaker.tuner import HyperparameterTuner, IntegerParameter, CategoricalParameter, ContinuousParameter
hyperparameter_ranges = {'optimizer': CategoricalParameter(['sgd', 'Adam']),
                         'learning_rate': ContinuousParameter(0.01, 0.2),
                         'num_epoch': IntegerParameter(10, 50)}

The tuning job will select parameters from these ranges and use those to determine the best place to focus training efforts. There are few types of parameters:

  • Categorical parameters use one value from a discrete set.
  • Continuous parameters can use any real number value between the minimum and maximum value.
  • Integer parameters can use any integer within the bounds specified.

Now that we have our ranges defined we want to define our success metric and a regular expression for finding that metric in the training job logs.


objective_metric_name = 'Validation-accuracy'
metric_definitions = [{'Name': 'Validation-accuracy',
                       'Regex': 'Validation-accuracy=([0-9\.]+)'}]

Now, with just these few things defined we can start our tuning job!


tuner = HyperparameterTuner(estimator,
                            objective_metric_name,
                            hyperparameter_ranges,
                            metric_definitions,
                            max_jobs=9,
                            max_parallel_jobs=3)
tuner.fit({'train': train_data_location, 'test': test_data_location})

Now, we can open up the SageMaker console, select the Hyperparameter tuning jobs sub-console and check out all our tuning jobs.

We can click on the job we just created to get some more detail and explore the results of the tuning.

By default the console will show us the best job and the parameters used but we can also check out each of the other jobs.

Hopping back over to our notebook instance, we have a handy analytics object from tuner.analytics() that we can use to visualize the results of the training with some bokeh plots. Some examples of this are provided in the SageMaker example notebooks.

This feature works for built-in algorithms, jobs created with the SageMaker Python SDK, or even bring-your-own training jobs in docker.

We can even create tuning jobs right in the console by clicking Create hyperparameter tuning job.

First we select a name for our job, an IAM role and which VPC it should run in, if any.

Next, we configure the training job. We can use built-in algorithms or a custom docker image. If we’re using a custom image this would be where we defined the regex to to find the objective metric in the logs. For now we’ll just select XGBoost and click next.

Now we’ll configure our tuning job parameters just like in the notebook example. I’ll select the area under the curve (AUC) as the objective metric to optimize. Since this is a builtin algorithm the regex for that metric was already filled in by the previous step. I’ll set the minimum and maximum number of rounds and click next.

In the next screen we can configure the input channels that our algorithm is expecting as well as the location to output the models. We’d typically have more than just the “train” channel and would have an “eval” channel as well.

Finally, we can configure the resource limits for this tuning job.

Now we’re off to the races tuning!

Additional Resources

To take advantage of automatic model tuning there are really only a few things users have to define: the hyperparameter ranges, the success metric and a regex to find it, the number of jobs to run in parallel, and the maximum number of jobs to run. For the built-in algorithms we don’t even need to define the regex. There’s a small trade-off between the number of parallel jobs used and the accuracy of the final model. Increasing max_parallel_jobs will cause the tuning job to finish much faster but a lower parallelism will generally provide a slightly better final result.

Amazon SageMaker Automatic Model Tuning is provided at no additional charge, you pay only for the underlying resources used by the training jobs that the tuning job launches. This feature is available now in all regions where SageMaker is available. This feature is available in the API and training jobs launched by automatic model tuning are visible in the console. You can find our more by reading the documentation.

I really think this feature will save developers a lot of time and effort and I’m excited to see what customers do with it. As always, we welcome your feedback in the comments or on Twitter!

Randall

Amazon EKS – Now Generally Available

We announced Amazon Elastic Container Service for Kubernetes and invited customers to take a look at a preview during re:Invent 2017. Today I am pleased to be able to let you know that Amazon EKS is available for use in production form. It has been certified as Kubernetes conformant, and is ready to run your existing Kubernetes workloads.

Based on the most recent data from the Cloud Native Computing Foundation, we know that AWS is the leading environment for Kubernetes, with 57% of all companies who run Kubernetes choosing to do so on AWS. Customers tell us that Kubernetes is core to their IT strategy, and are already running hundreds of millions of containers on AWS every week. Amazon EKS simplifies the process of building, securing, operating, and maintaining Kubernetes clusters, and brings the benefits of container-based computing to organizations that want to focus on building applications instead of setting up a Kubernetes cluster from scratch.

AWS Inside
Amazon EKS takes advantage of the fact that it is running in the AWS Cloud, making great use of many AWS services and features, while ensuring that everything you already know about Kubernetes remains applicable and helpful. Here’s an overview:

Multi-AZ – The Kubernetes control plane (the API server and the etcd database) are run in high-availability fashion across three AWS Availability Zones. Master nodes are monitored and replaced if they fail, and are also patched and updated automatically.

IAM IntegrationAmazon EKS uses the Heptio Authenticator for authentication. You can make use of IAM roles and avoid the pain that comes with managing yet another set of credentials.

Load Balancer Support – You can route traffic to your worker nodes using the AWS Network Load Balancer, the AWS Application Load Balancer, or the original (classic) Elastic Load Balancer.

EBS – Kubernetes PersistentVolumes (used for cluster storage) are implemented as Amazon Elastic Block Store (EBS) volumes.

Route 53 – The External DNS project allows services in Kubernetes clusters to be accessed via Route 53 DNS records. This simplifies service discovery and supports load balancing.

Auto Scaling – Your clusters can make use of Auto Scaling, growing and shrinking in response to changes in load.

Container Interface – The Container Network Interface for Kubernetes uses Elastic Network Interfaces to provide static IP addresses for Kubernetes Pods.

For a more detailed look at these features, read about Amazon Elastic Container Service for Kubernetes.

Amazon EKS is built around a shared-responsibility model; the control plane nodes are managed by AWS and you run the worker nodes. This gives you high availability and simplifies the process of moving existing workloads to EKS. Here’s a very high-level overview:

 

Creating an Amazon EKS Cluster
To create a cluster, I provision the control plane, provision and connect the worker cluster, and launch my containers. In the example below I will create a new VPC for my worker cluster, but I can also use an existing one, as long as the desired subnets are tagged with the name of my Kubernetes cluster.

Following the directions in the Amazon EKS Getting Started Guide, I begin by creating an IAM role. Kubernetes assumes this role and uses it to create AWS resources such as Elastic Load Balancers. Once created, this role can be used for all of my clusters. I simply create a CloudFormation stack using the template referred to in the Getting Started Guide:

I acknowledge that the stack will create a role, and click Create to proceed:

The role is created in seconds, and the ARN is shown in the stack’s Output tab (I’ll need it later):

Next, I create a VPC (Virtual Private Cloud) using the sample template from the Getting Started Guide, with the following parameters:

The template creates a VPC that has two subnets, along with all of the necessary route tables, gateways, and security groups):

As is the case with the ARN, I will need the ID of the security group later.

Next, I download kubectl and set it up to use the Heptio Authenticator. The authenticator allows kubectl to make use of IAM authentication when it accesses my Kubernetes clusters. Instructions for downloading and setup are in the Getting Started Guide and I follow them as directed.

To wrap up the setup process, I ensure that I am running the latest version of the AWS Command Line Interface (CLI) (If I was running an older version, the eks command would not be available):

With my IAM role, my VPC, and my tooling all in place, I am ready to create my first Amazon EKS cluster!

I log in to the EKS Console using an IAM user that has administrative privileges (root credentials cannot be used due to the way that the Heptio Authenticator works) and click Create cluster:

I enter a name for my cluster (which must match the one that I entered when I created the VPC, because Kubernetes relies on tagging of subnets), along with the subnet IDs and the security group ID, both for the VPC, and click Create:

My control plane cluster starts out in CREATING status, and transitions to ACTIVE in 10 minutes or less:

Now I need to configure kubectl so that it can access my cluster. Before I can do this, I need to use the CLI to retrieve the certificate authority data:

$ aws eks describe-cluster --region us-west-2 --cluster-name jeff1 --query cluster.certificateAuthority.data

This command returns a long string of data that I’ll need in a minute.

I also retrieve the cluster endpoint from the console:

I make sure that I am in my home directory, create sub-directory .kube, and create file config-jeff1 in it. Then I open config-jeff1 in my editor, copy the templated config file from the Getting Started Guide and finalize the cluster endpoint, certificate, and cluster name. My file looks like this:

apiVersion: v1
clusters:
- cluster:
    server: https://FDA1964D96C9EEF2B76684C103F31C67.sk1.us-west-2.eks.amazonaws.com
    certificate-authority-data: "...."
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: aws
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: heptio-authenticator-aws
      args:
        - "token"
        - "-i"

Before I test kubectl, I need to ensure that my CLI is configured to use the same IAM user that I used when I logged in to the console to create the cluster:

And now I can run a quick test to verify that everything is working as expected:

At this point I have set up my master VPC and my Kubernetes control plane. I’m ready to create some worker nodes (EC2 instances). Once again, this is done using a CloudFormation template:

The stack is created in a couple of minutes and sets up IAM roles, security groups, and auto scaling:

Now I need to set up a configurator map so that the worker nodes know how to join the cluster. I download the map, add the ARN of the NodeInstanceRole from the stack, and apply the configuration:

Then I check and see that my nodes are ready:

Running the Guest Book Sample
My Kubnernetes cluster is all set and I can use the Guest Book application to test it out. I create the Kubernetes replication controllers and services:

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/kubernetes/v1.10.0/examples/guestbook-go/redis-master-controller.json
replicationcontroller "redis-master" created
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/kubernetes/v1.10.0/examples/guestbook-go/redis-master-service.json
service "redis-master" created
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/kubernetes/v1.10.0/examples/guestbook-go/redis-slave-controller.json
replicationcontroller "redis-slave" created
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/kubernetes/v1.10.0/examples/guestbook-go/redis-slave-service.json
service "redis-slave" created
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/kubernetes/v1.10.0/examples/guestbook-go/guestbook-controller.json
replicationcontroller "guestbook" created
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/kubernetes/v1.10.0/examples/guestbook-go/guestbook-service.json
service "guestbook" created

I list the running services and capture the external IP address & port:

and visit the address in my web browser:

Things to Know
We make upstream contributions to the Kubernetes repo and to projects such as the CNI Plugin, the Heptio AWS Authenticator, and Virtual Kubelet. We are currently looking for Systems Development Engineers, DevOps Engineers, Product Managers, and Solution Architects with Kubernetes experience; check out the full list of open positions to learn more.

Amazon EKS is available today in the US East (N. Virginia) and US West (Oregon) Regions and will be expanding to others very soon. We have a detailed roadmap and plan to crank out plenty of additional features this year.

You pay $0.20 per hour for the EKS Control Plane, and usual EC2, EBS, and Load Balancing prices for resources that run in your account. See the EKS Pricing page for more information.

Jeff;

 

SAP on AWS – Past, Present, and Future

While many of my AWS colleagues are preparing for SAPPHIRE NOW, I thought this would be a good time to bring you up to date on what we have already done to make AWS a great home for SAP’s products and to share our plans to make it even better.

The Story So Far
Our enterprise customers want to bring gigantic, memory-intensive workloads to the AWS Cloud with a special focus on large-scale production deployments of SAP HANA. Here’s what we have done so far to meet this important requirement:

May 2016 – We announced the x1.32xlarge instance type with 2 TB of memory, purpose-built for running SAP HANA in the cloud.

August 2016 – We announced SAP certification and support for scale-out clusters of up to 7 nodes and 14 TB of memory.

October 2016 – We announced the x1.16xlarge instance type with 1 TB of memory, perfect for testing and for smaller SAP HANA deployments, along with increased regional availability for both of the X1 instances.

May 2017 – We announced the x1e.32xlarge instance type with 4 TB of memory and SAP support for very large scale-out SAP HANA clusters of up to 17 nodes (34 TB of memory).

November 2017 – We announced SAP support for even larger on-demand SAP HANA clusters with up to 25 x1.32xlarge nodes (50 TB of memory).

Along the way, we have been working with customers like Brooks Brothers, Visy, Sumitomo Chemicals, and Kellogg’s to build business-critical HANA implementations on AWS. These customers (and many others) have improved their agility, realized cost savings, and increased performance as part of their move to the cloud.

Right Here, Right Now
As you may know, the C5 and M5 instances are powered by the latest Intel® Xeon® Scalable (Skylake) processors, and make use of our new lightweight, hardware-accelerated Nitro hypervisor. Both types of instances are fully certified by SAP, and deliver a measurable performance increase with respect to their predecessors. The Nitro Hypervisor provides consistent performance and increased compute and memory resources for virtualized EC2 instances by removing host system software components. It allows us to offer larger instance sizes (like c5.18xlarge) that make just about all of the server’s resources available to customers.

As an indication of our progress over the last couple of years, our first SAP certified NetWeaver installations on m2.4xlarge instances delivered 7400 SAPS (925 per vCPU). Today, the m5.24xlarge instances can deliver 135,230 SAPS (1409 per vCPU), our best performance to date. You can read the new SAP benchmarks for C5 and M5 instances, along with Measuring in SAPS, to learn more.

In the Works – Instances with More Memory
Our collaboration with SAP began in 2008 with the goal of providing our customers with options for running their mission-critical SAP systems in the cloud. We worked side-by-side with SAP to enable production deployments of HANA in 2014, and now offer a wide range of EC2 instances that are certified by SAP to run HANA.

Our goal is to make it as easy as possible to run HANA and to provide you with instance sizes that are a great fit for many different applications and installations. At the last SAPPHIRE NOW conference, we announced our plans to launch EC2 instances with 8 TB to 16 TB of memory. Today I would like to tell you a bit more about the specs and sizes for these instances.

We are planning to launch high-memory EC2 Bare Metal instances with 6 TB, 9 TB, and 12 TB of memory, designed from the ground up to run mission-critical deployments of SAP HANA. Like the existing Bare Metal instances, these instances allow the operating system to run directly on the underlying hardware while still providing access to all of the benefits of the cloud as full-fledged members of the EC2 family.

The instances run on an 8-socket platform built with Intel Xeon Scalable (Skylake) processors. They can be launched in a VPC, offer ENA-based Enhanced Networking and EBS-optimization by default, and are available on EC2 Dedicated Hosts. You will be able to launch them in all of the usual ways, and to use IAM to control authentication, authorization, and auditing. Instances will be able to make use of multiple EBS volumes, each storing up to 16 TB of data, for elastic capacity.

I did not have the opportunity to go hands-on with the new instances, but my colleagues shared a few screen shots with me! Here’s some of the output from dmesg on an instance with 6 TB of memory:

And here’s what lscpu displays:

We plan to make these instances available in private preview this summer, and to move them to general availability this fall. While 12 TB instances are certainly a big step forward, we don’t plan to stop there, and are working on even bigger ones — instances with more than 16 TB of memory are in the works as well!

If you would like to join the private preview for these new instances, please contact us.

Amazon AppStream 2.0 with SAP GUI
In other AWS / SAP news, you can now use Amazon AppStream 2.0 to visualize the SAP GUI in any browser that is HTML5-compatible.

This is a clean, simple, and efficient alternative to installing the SAP GUI on every desktop. Response time improves, as does user productivity, because less data moves between client and server. Replacing hundreds or thousands of installed copies of SAP GUI with a centrally managed image also reduces the overall management effort.

To learn more about this cool new way to make the SAP GUI available to your users, read Deploying SAP GUI on Amazon AppStream 2.0.

Say Hello at SAPPHIRE NOW
The AWS team will be in booth 642 at SAPPHIRE this week with a full set of sessions from our team, our customers, and our partners in our in-booth theater. Many of our customers will also be telling their stories during sessions throughout the event. A listing of available sessions and activities can be found here.

Jeff;

EC2 Instance Update – M5 Instances with Local NVMe Storage (M5d)

Earlier this month we launched the C5 Instances with Local NVMe Storage and I told you that we would be doing the same for additional instance types in the near future!

Today we are introducing M5 instances equipped with local NVMe storage. Available for immediate use in 5 regions, these instances are a great fit for workloads that require a balance of compute and memory resources. Here are the specs:

Instance Name vCPUs RAM Local Storage EBS-Optimized Bandwidth Network Bandwidth
m5d.large 2 8 GiB 1 x 75 GB NVMe SSD Up to 2.120 Gbps Up to 10 Gbps
m5d.xlarge 4 16 GiB 1 x 150 GB NVMe SSD Up to 2.120 Gbps Up to 10 Gbps
m5d.2xlarge 8 32 GiB 1 x 300 GB NVMe SSD Up to 2.120 Gbps Up to 10 Gbps
m5d.4xlarge 16 64 GiB 1 x 600 GB NVMe SSD 2.210 Gbps Up to 10 Gbps
m5d.12xlarge 48 192 GiB 2 x 900 GB NVMe SSD 5.0 Gbps 10 Gbps
m5d.24xlarge 96 384 GiB 4 x 900 GB NVMe SSD 10.0 Gbps 25 Gbps

The M5d instances are powered by Custom Intel® Xeon® Platinum 8175M series processors running at 2.5 GHz, including support for AVX-512.

You can use any AMI that includes drivers for the Elastic Network Adapter (ENA) and NVMe; this includes the latest Amazon Linux, Microsoft Windows (Server 2008 R2, Server 2012, Server 2012 R2 and Server 2016), Ubuntu, RHEL, SUSE, and CentOS AMIs.

Here are a couple of things to keep in mind about the local NVMe storage on the M5d instances:

Naming – You don’t have to specify a block device mapping in your AMI or during the instance launch; the local storage will show up as one or more devices (/dev/nvme*1 on Linux) after the guest operating system has booted.

Encryption – Each local NVMe device is hardware encrypted using the XTS-AES-256 block cipher and a unique key. Each key is destroyed when the instance is stopped or terminated.

Lifetime – Local NVMe devices have the same lifetime as the instance they are attached to, and do not stick around after the instance has been stopped or terminated.

Available Now
M5d instances are available in On-Demand, Reserved Instance, and Spot form in the US East (N. Virginia), US West (Oregon), EU (Ireland), US East (Ohio), and Canada (Central) Regions. Prices vary by Region, and are just a bit higher than for the equivalent M5 instances.

Jeff;

 

AWS Online Tech Talks – June 2018

AWS Online Tech Talks – June 2018

Join us this month to learn about AWS services and solutions. New this month, we have a fireside chat with the GM of Amazon WorkSpaces and our 2nd episode of the “How to re:Invent” series. We’ll also cover best practices, deep dives, use cases and more! Join us and register today!

Note – All sessions are free and in Pacific Time.

Tech talks featured this month:

 

Analytics & Big Data

June 18, 2018 | 11:00 AM – 11:45 AM PTGet Started with Real-Time Streaming Data in Under 5 Minutes – Learn how to use Amazon Kinesis to capture, store, and analyze streaming data in real-time including IoT device data, VPC flow logs, and clickstream data.
June 20, 2018 | 11:00 AM – 11:45 AM PT – Insights For Everyone – Deploying Data across your Organization – Learn how to deploy data at scale using AWS Analytics and QuickSight’s new reader role and usage based pricing.

 

AWS re:Invent
June 13, 2018 | 05:00 PM – 05:30 PM PTEpisode 2: AWS re:Invent Breakout Content Secret Sauce – Hear from one of our own AWS content experts as we dive deep into the re:Invent content strategy and how we maintain a high bar.
Compute

June 25, 2018 | 01:00 PM – 01:45 PM PTAccelerating Containerized Workloads with Amazon EC2 Spot Instances – Learn how to efficiently deploy containerized workloads and easily manage clusters at any scale at a fraction of the cost with Spot Instances.

June 26, 2018 | 01:00 PM – 01:45 PM PTEnsuring Your Windows Server Workloads Are Well-Architected – Get the benefits, best practices and tools on running your Microsoft Workloads on AWS leveraging a well-architected approach.

 

Containers
June 25, 2018 | 09:00 AM – 09:45 AM PTRunning Kubernetes on AWS – Learn about the basics of running Kubernetes on AWS including how setup masters, networking, security, and add auto-scaling to your cluster.

 

Databases

June 18, 2018 | 01:00 PM – 01:45 PM PTOracle to Amazon Aurora Migration, Step by Step – Learn how to migrate your Oracle database to Amazon Aurora.
DevOps

June 20, 2018 | 09:00 AM – 09:45 AM PTSet Up a CI/CD Pipeline for Deploying Containers Using the AWS Developer Tools – Learn how to set up a CI/CD pipeline for deploying containers using the AWS Developer Tools.

 

Enterprise & Hybrid
June 18, 2018 | 09:00 AM – 09:45 AM PTDe-risking Enterprise Migration with AWS Managed Services – Learn how enterprise customers are de-risking cloud adoption with AWS Managed Services.

June 19, 2018 | 11:00 AM – 11:45 AM PTLaunch AWS Faster using Automated Landing Zones – Learn how the AWS Landing Zone can automate the set up of best practice baselines when setting up new

 

AWS Environments

June 21, 2018 | 11:00 AM – 11:45 AM PTLeading Your Team Through a Cloud Transformation – Learn how you can help lead your organization through a cloud transformation.

June 21, 2018 | 01:00 PM – 01:45 PM PTEnabling New Retail Customer Experiences with Big Data – Learn how AWS can help retailers realize actual value from their big data and deliver on differentiated retail customer experiences.

June 28, 2018 | 01:00 PM – 01:45 PM PTFireside Chat: End User Collaboration on AWS – Learn how End User Compute services can help you deliver access to desktops and applications anywhere, anytime, using any device.
IoT

June 27, 2018 | 11:00 AM – 11:45 AM PTAWS IoT in the Connected Home – Learn how to use AWS IoT to build innovative Connected Home products.

 

Machine Learning

June 19, 2018 | 09:00 AM – 09:45 AM PTIntegrating Amazon SageMaker into your Enterprise – Learn how to integrate Amazon SageMaker and other AWS Services within an Enterprise environment.

June 21, 2018 | 09:00 AM – 09:45 AM PTBuilding Text Analytics Applications on AWS using Amazon Comprehend – Learn how you can unlock the value of your unstructured data with NLP-based text analytics.

 

Management Tools

June 20, 2018 | 01:00 PM – 01:45 PM PTOptimizing Application Performance and Costs with Auto Scaling – Learn how selecting the right scaling option can help optimize application performance and costs.

 

Mobile
June 25, 2018 | 11:00 AM – 11:45 AM PTDrive User Engagement with Amazon Pinpoint – Learn how Amazon Pinpoint simplifies and streamlines effective user engagement.

 

Security, Identity & Compliance

June 26, 2018 | 09:00 AM – 09:45 AM PTUnderstanding AWS Secrets Manager – Learn how AWS Secrets Manager helps you rotate and manage access to secrets centrally.
June 28, 2018 | 09:00 AM – 09:45 AM PTUsing Amazon Inspector to Discover Potential Security Issues – See how Amazon Inspector can be used to discover security issues of your instances.

 

Serverless

June 19, 2018 | 01:00 PM – 01:45 PM PTProductionize Serverless Application Building and Deployments with AWS SAM – Learn expert tips and techniques for building and deploying serverless applications at scale with AWS SAM.

 

Storage

June 26, 2018 | 11:00 AM – 11:45 AM PTDeep Dive: Hybrid Cloud Storage with AWS Storage Gateway – Learn how you can reduce your on-premises infrastructure by using the AWS Storage Gateway to connecting your applications to the scalable and reliable AWS storage services.
June 27, 2018 | 01:00 PM – 01:45 PM PTChanging the Game: Extending Compute Capabilities to the Edge – Discover how to change the game for IIoT and edge analytics applications with AWS Snowball Edge plus enhanced Compute instances.
June 28, 2018 | 11:00 AM – 11:45 AM PTBig Data and Analytics Workloads on Amazon EFS – Get best practices and deployment advice for running big data and analytics workloads on Amazon EFS.