My content goes here! Decades ago, I wrote page after page of code in 6502 assembly language. After assembling and linking the code, I would load it into memory, set breakpoints at strategic locations, and step through to make sure that everything worked as intended. These days, I no longer have the opportunity to write or debug any non-trivial code, so I was a bit apprehensive when it came time to write this blog post (truth be told, I have been procrastinating for several weeks).

SAM CLI
I want to tell you about the new Serverless Application Model (SAM) Command Line Interface, and to gain some confidence in my ability to build something using AWS Lambda as I do so! Let’s review some terms to get started:

AWS SAM, short for Serverless Application Model, is an open source framework you can use to build serverless applications on AWS. It provides a shorthand syntax you can use to describe your application (Lambda functions, API endpoints, DynamoDB tables, and other resources) using a simple YAML template. During deployment, SAM transforms and expands the shorthand SAM syntax into an AWS CloudFormation template. Then, CloudFormation provisions your resources in a reliable and repeatable fashion.

The AWS SAM CLI, formerly known as SAM Local, is a command-line interface that supports building SAM-based applications. It supports local development and testing, and is also an active open source project. The CLI lets you choose between Python, Node, Java, Go, .NET, and includes a healthy collection of templates to help get you started.

The sam local command in the SAM CLI, delivers support for local invocation and testing of Lambda functions and SAM-based serverless applications, while running your function code locally in a Lambda-like execution environment. You can also use the sam local command to generate sample payloads locally, start a local endpoint to test your APIs, or automate testing of your Lambda functions.

Installation and Setup
Before I can show you how to use the SAM CLI, I need to install a couple of packages. The functions provided by sam local make use of Docker, so I need to work in a non-virtualized environment for a change! Here’s an overview of the setup process:

Docker – I install the Community Edition of Docker for Windows (a 512 MB download), and run docker ps to verify that it is working:

Python – I install Python 3.6 and make sure that it is on my Windows PATH:

Visual Studio Code – I install VS Code and the accompanying Python Extension.

AWS CLI – I install the AWS CLI:

And configure my credentials:

SAM – I install the AWS SAM CLI using pip:

Now that I have all of the moving parts installed, I can start to explore SAM.

Using SAM CLI
I create a directory (sam_apps) for my projects, and then I run sam init to create my first project:

This creates a sub-directory (sam-app) with all of the necessary source and configuration files inside:

I create a build directory inside of hello_world, and then I install the packages defined in requirements. The build directory contains the source code and the Python packages that are loaded by SAM Local:

And one final step! I need to copy the source files to the build directory in order to deploy them:

My app (app.py and an empty __init__.py) is ready to go, so I start up a local endpoint:

At this point, the endpoint is listening on port 3000 for an HTTP connection, and a Docker container will launch when the connection is made. The build directory is made available to the container so that the Python packages can be loaded and the code in app.py run.

When I open http://127.0.0.1:3000/hello in my browser, the container image is downloaded if necessary, the code is run, and the output appears in my browser:

Here’s what happens on the other side. You can see all of the important steps here, including the invocation of the code, download of the image, mounting the build directory in the container, and the request logging:

I can modify the code, refresh the browser tab, and the new version is run:

The edit/deploy/test cycle is incredibly fast, and you will be more productive than ever!

There is one really important thing to remember here. The initial app.py file was created in the hello_world directory, and I copied it to the build directory a few steps ago. I can do this deployment step each time, or I can simply decide that the code in the build directory is the real deal and edit it directly. This will affect my source code control plan once I start to build and version my code.

What’s Going On
Now that the sample code is running, let’s take a look at the SAM template (imaginatively called template.yaml). In the interest of space, I’ll skip ahead to the Resources section:

Resources:

    HelloWorldFunction:
        Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
        Properties:
            CodeUri: hello_world/build/
            Handler: app.lambda_handler
            Runtime: python3.6
            Environment: # More info about Env Vars: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#environment-object
                Variables:
                    PARAM1: VALUE
            Events:
                HelloWorld:
                    Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
                    Properties:
                        Path: /hello
                        Method: get

This section defines the HelloWorldFunction, indicates where it can be found (hello_world/build/), how to run it (python3.6), and allows environment variables to be defined and set. Then it indicates that the function can process the HelloWorld event, which is generated by a GET on the indicated path (/hello).

This template is not reloaded automatically; if I change it I will need to restart SAM Local. I recommend that you spend some time altering the names and paths here and watching the errors that arise. This will give you a good understanding of what is happening behind the scenes, and will improve your productivity later.

The remainder of the template describes the outputs from the template (the API Gateway endpoint, the function’s ARN, and the function’s IAM Role). These values do not affect local execution, but are crucial to a successful cloud deployment.

Outputs:

    HelloWorldApi:
      Description: "API Gateway endpoint URL for Prod stage for Hello World function"
      Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"

    HelloWorldFunction:
      Description: "Hello World Lambda Function ARN"
      Value: !GetAtt HelloWorldFunction.Arn

    HelloWorldFunctionIamRole:
      Description: "Implicit IAM Role created for Hello World function"
      Value: !GetAtt HelloWorldFunctionRole.Arn

You can leave all of these as-is until you have a good understanding of what’s going on.

Debugging with SAM CLI and VS Code
Ok, now let’s get set up to do some interactive debugging! This took me a while to figure out and I hope that you can benefit from my experience. The first step is to install the ptvsd package:

Then I edit requirements.txt to indicate that my app requires ptvsd (I copied the version number from the package name above):

requests==2.18.4
ptvsd==4.1.4

Next, I rerun pip to install this new requirement in my build directory:

Now I need to modify my code so that it can be debugged. I add this code after the existing imports:

import ptvsd
ptvsd.enable_attach(address=('0.0.0.0', 5858), redirect_output=True)
ptvsd.wait_for_attach()

The first statement tells the app that the debugger will attach to it on port 5858; the second pauses the code until the debugger is attached (you could make this conditional).

Next, I launch VS Code and select the root folder of my application:

Now I need to configure VS Code for debugging. I select the debug icon, click the white triangle next to DEBUG, and select Add Configuration:

I select the Python configuration, replace the entire contents of the file (launch.json) with the following text, and save the file (File:Save).

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [

        {
            "name": "Debug with SAM CLI (Remote Debug)",
            "type": "python",
            "request": "attach",
            "port": 5858,
            "host":  "localhost",
            "pathMappings": [
                {
                "localRoot": "${workspaceFolder}/hello_world/build",
                "remoteRoot" : "/var/task"
                }
            ]
        }
    ]
}

Now I choose this debug configuration from the DEBUG menu:

Still with me? We’re almost there!

I start SAM Local again, and tell it to listen on the debug port:

I return to VS Code and set a breakpoint (good old F9) in my code:

One thing to remember — be sure to open app.py in the build directory and set the breakpoint there.

Now I return to my web browser and visit the local address (http://127.0.0.1:3000/hello) again. The container starts up to handle the request and it runs app.py. The code runs until it hits the call to wait_for_attach, and now I hit F5 in VS Code to start debugging.

The breakpoint is hit, I single-step across the requests.get call, and inspect the ip variable:

Then I hit F5 to continue, and the web request completes. As you can see, I can use the full power of the VS Code debugger to build and debug my Lambda functions. I’ve barely scratched the surface here, and encourage you to follow along and pick up where I left off. To learn more, read Test Your Serverless Applications Locally Using SAM CLI.

Cloud Deployment
The SAM CLI also helps me to package my finished code, upload it to S3, and run it. I start with an S3 bucket (jbarr-sam) and run sam package. This creates a deployment package and uploads it to S3:

This takes a few seconds. Then I run sam deploy to create a CloudFormation stack:

If the stack already exists, SAM CLI will create a Change Set and use it to update the stack. My stack is ready in a minute or two, and includes the Lambda function, an API Gateway, and all of the supporting resources:

I can locate the API Gateway endpoint in the stack outputs:

And access it with my browser, just like I did when the code was running locally:

I can also access the CloudWatch logs for my stack and function using sam logs:

My SAM apps are now visible in the Lambda Console (this is a relatively new feature):

I can see the template and the app’s resources at a glance:

And I can see the relationship between resources:

There’s also a monitoring dashboard:

I can customize the dashboard by adding an Amazon CloudWatch dashboard to my template (read Managing Applications in the AWS Lambda Console to learn more).

That’s Not All
Believe it or not, I have given you just a taste of what you can do with SAM, SAM CLI, and the sam local command. Here are a couple of other cool things that you should know about:

Local Function Invocation – I can directly invoke Lambda functions:

Sample Event Source Generation – If I am writing Lambda functions that respond to triggers from other AWS services (S3 PUTs and so forth), I can generate sample events and use them to invoke my functions:

In a real-world situation I would redirect the output to a file, make some additional customization if necessary, and then use it to invoke my function.

Cookiecutter Templates – The SAM CLI can use Cookiecutter templates to create projects and we have created several examples to get you started. Take a look at Cookiecutter AWS Sam S3 Rekognition Dynamodb Python and Cookiecutter for AWS SAM and .NET to learn more.

CloudFormation Extensions – AWS SAM extends CloudFormation and lets you benefit from the power of infrastructure as code. You get reliable and repeatable deployments and the power to use the full suite of CloudFormation resource types, intrinsic functions, and other template features.

Built-In Best Practices – In addition to the benefits that come with an infrastructure as code model, you can easily take advantage of other best practices including code reviews, safe deployments through AWS CodePipeline, and tracing using AWS X-Ray.

Deep Integration with Development Tools – You can use AWS SAM with a suite of AWS tools for building serverless applications. You can discover new applications in the AWS Serverless Application Repository. For authoring, testing, and debugging SAM-based serverless applications, you can use the AWS Cloud9 IDE. To build a deployment pipeline for your serverless applications, you can use AWS CodeBuild, AWS CodeDeploy, and AWS CodePipeline. You can also use AWS CodeStar to get started with a project structure, code repository, and a CI/CD pipeline that’s automatically configured for you. To deploy your serverless application you can use the AWS SAM Jenkins plugin, and you can use Stackery.io’s toolkit to build production-ready applications.

Check it Out
I hope that you have enjoyed this tour, and that you can make good use of SAM in your next serverless project!

— Jeff;

test blog goes here

Last year we launched new AWS Regions in France and China (Ningxia), and announced that we are working on regions in Bahrain, Hong Kong SAR, Sweden, and a second GovCloud Region in the United States.

South Africa in Early 2020
Today, I am happy to announce that we will be opening an AWS Region in South Africa in the first half of 2020. The new Region will be based in Cape Town, will be comprised of three Availability Zones, and will give AWS customers and partners the ability to run their workloads and store their data in South Africa. The addition of the AWS Africa (Cape Town) Region will also enable organizations to provide lower latency to end users across Sub-Saharan Africa and will enable more African organizations to leverage advanced technologies such as Artificial Intelligence, Machine Learning, Internet of Things (IoT), mobile services, and more to drive innovation.

AWS customers are already making use of 55 Availability Zones across 19 infrastructure regions worldwide. Today’s announcement brings the total number of global regions (operational and in the works) up to 23.

A Growing Presence
The new Region is the latest of a series of investments in South Africa, and is part our commitment to support South Africa’s transformation. In 2004, Amazon opened a Development Center in Cape Town that focuses on building pioneering networking technologies, next generation software for customer support, and the technology behind Amazon EC2. AWS has also added a number of teams including account managers, customer services reps, partner managers, solutions architects, and more, helping customers of all sizes as they move to the cloud.

In 2015, we continued our expansion, opening an office in Johannesburg, and in 2017 we brought the Amazon Global Network to Africa through AWS Direct Connect. Earlier this year we launched infrastructure on the African continent introducing Amazon CloudFront to South Africa, with two new edge locations in Cape Town and Johannesburg. We also support the growth of technology education with AWS Academy and AWS Educate and have supported the growth of new businesses through AWS Activate in the country for many years.

The addition of the AWS Region in South Africa will help builders and entrepreneurs in enterprises, the public sector, and startups across Sub-Saharan Africa to innovate and grow their organizations.

Talk to Us
As always, we are looking forward to serving new and existing customers in South Africa and working with partners across the region. Of course, the new Region will also be open to existing AWS customers who would like to serve users in South Africa and across the African continent.

To learn more about the AWS South Africa Region feel free to contact our team at aws-in-africa@amazon.com. If you are interested in joining the team and would like to learn more about AWS positions in South Africa, take a look at the Amazon Jobs site.

— Jeff;

The blog post that we published over a decade ago to launch the Simple Monthly Calculator still shows up on our internal top-10 lists from time to time! Since that post was published, we have extended, redesigned, and even rebuilt the calculator a time or two.

New Calculator
Starting with a blank screen, an empty code repo, and plenty of customer feedback, we are building a brand-new AWS Pricing Calculator. The new calculator is designed to help you estimate and understand your eventual AWS costs. We did our best to avoid excessive jargon and to make the calculations obvious, transparent, and accessible. You can see the options that are available to you, explore the associated costs, and make high-quality data-driven decisions.

We’re starting out with support for EC2 instances, EBS volumes, and a very wide variety of purchasing models, with plans to add support for more services as quickly as possible.

A Quick Tour
The new calculator lives at https://calculator.aws . Each estimate consists of one or more groups and the first one is created automatically:

Each group has a name, and has pricing for services in a particular AWS region. I click Edit group to change the name and pick a region, and click Apply:

Back at the main page of the calculator, I click Add service and choose to configure some EC2 instances. The group can contain multiple types and configurations of instances; I click Configure to move ahead:

At this point I can make a Quick estimate (the default), or supply more details as part of an Advanced estimate. I’ll start with a Quick estimate:

Here are a couple of things to keep in mind when you make a quick estimate:

Instance Type – I have two options for choosing EC2 instance types; I can enter my resource requirements (vCPU count, memory size, and GPU count) and have the calculator choose the option with the lowest price, or I can pick an EC2 instance type by name.

Pricing Strategy – I can choose to use On-Demand Instances, Convertible Reserved Instances, or Standard Reserved Instances, and can choose payment terms and options for RI’s.

EBS Volumes – I can choose the type and size of an EBS volume for the instance. Right now, the calculator allows you to associate one volume with each EC2 instance. If you need more than one, specify the total amount of storage you need across all volumes.

Details – I can expand the Show calculation section to see the math:

After I have made my choices, I click Add to my estimate to move ahead. My selections, along with their costs (annual, upfront, and monthly), are displayed:

I can go back and add another service, or create another group. I’ll add another EC2 instance, using an Advanced estimate this time around. Here’s where I start:

I have very fine-grained control over each aspect of my estimate. For example, I can characterize my workload in great detail. I click on Workload, and have the ability to select the graph that best represents my monthly workload:

I can even model workloads that have two or more independent daily (in this case) spike patterns. As I refine my model, the calculator figures out the most economical combination of On-Demand and Reserved Instances, and shows me the results:

The calculations are driven by the selection in the Pricing strategy. The default value, and the one that I used for the previous screen shot, is Cost optimized. I have other choices as well:

I can also model my data transfer in, out, and to other AWS regions:

Once I am happy with the results I click Add to my estimate, and take a look at my selections and their prices:

I can click Export to capture my estimate in spreadsheet form:

Here’s the data (I hid a few columns for clarity):

As you can see, the new calculator will quickly become a useful part of your planning and decision-making process.

One important thing to keep in mind: your estimates are stored in state that is local to the browser tab, and will be lost if you close the tab. The team is already hard at work on features that will allow you to save and even share your estimates, for launch in early 2019.

Stay Tuned
We will be adding more services and more features to the calculator in the months to come, and I’ll share some updates with you from time to time, either in this blog or via Twitter. If you have ideas, complaints, or other feedback, don’t hesitate to click on the Feedback link at the top of the page.

— Jeff;

It is time for a quick Amazon RDS update. I’ve got lots of news to share:

Console Update – The RDS Console has a fresh, new look.

RDS Recommendations – You now get recommendations that will help you to configure your database instances per our best practices.

Performance Insights for MySQL – You can peer deep inside of MySQL and understand more about how your queries are processed.

M5 Instances – You can now use MySQL and MariaDB on M5 instances.

MySQL 8.0 – You can now use MySQL 8.0 in production form.

MariaDB 10.3 – You can now use MariaDB 10.3 in production form.

Let’s take a closer look…

Console Update
The RDS Console took on a fresh, new look earlier this year. We made it available to you in preview form during development, and it is now the standard experience for all AWS users. You can see an overview of your RDS resources at a glance, create a new database, access documentation, and more, all from the home page:

You also get direct access to Performance Insights and to the new RDS Recommendations.

RDS Recommendations
We want to make it easy for you to take our best practices in to account when you configure your RDS database instances, even as those practices improve. The new RDS Recommendations feature will periodically check your configuration, usage, and performance data and display recommended changes and improvements, focusing on performance, stability, and security. It works with all of the database engines, and is very easy to use. Open the RDS Console and click Recommendations to get started:

I can see all of the recommendations at a glance:

I can open a recommendation to learn more:

I have four options that I can take with respect to this recommendation:

Fix Immediately – I select some database instances and click Apply now.

Fix Later – I select some database instances and click Schedule for the next maintenance window.

Dismiss – I select some database instances and click Dismiss to indicate that I do not want to make any changes, and to acknowledge that I have seen the recommendation.

Defer – If I do nothing, the recommendations remain active and I can revisit them at another time.

Other recommendations may include other options, or might require me to take some other actions. For example, the procedure for enabling encryption depends on the database engine:

RDS Recommendations are available today at no charge in the US East (N. Virginia), US East (Ohio), US West (N. California), US West (Oregon), Europe (Ireland), Europe (Frankfurt), Asia Pacific (Tokyo), Asia Pacific (Sydney), and Asia Pacific (Singapore) Regions. We plan to add additional recommendations over time, and also expect to make the recommendations available via an API.

Performance Insights for MySQL
I can now peek inside of MySQL to see which queries, hosts, and users are consuming the most time, and why:

You can identify expensive SQL queries and other bottlenecks with a couple of clicks, looking back across the timeframe of your choice: an hour, a day, a week, or even longer.

This feature was first made available for PostgreSQL (both RDS and Aurora) and is now available for MySQL (again, both RDS and Aurora). To learn more, read Using Amazon RDS Performance Insights.

M5 Instances
The M5 instances deliver improved price/performance compared to M4 instances, and offer up to 10 Gbps of dedicated network bandwidth for database storage.

You can now launch M5 instances (including the new high-end m5.24xlarge) when using RDS for MySQL and RDS for MariaDB. You can scale up to these new instance types by modifying your existing DB instances:

MySQL 8
Version 8 of MySQL is now available on Amazon RDS. This version of MySQL offers better InnoDB performance, JSON improvements, better GIS support (new spatial datatypes, indexes, and functions), common table expressions to reduce query complexity, window functions, atomic DDLs for faster online schema modification, and much more (read the documentation to learn more).

MariaDB 10.3
Version 10.3 of MariaDB is now available on Amazon RDS. This version of MariaDB includes a new temporal data processing feature, improved Oracle compatibility, invisible columns, performance enhancements including instant ADD COLUMN operations & fast-fail DDL operations, and much more (read the documentation for a detailed list).

Available Now
All of the new features, engines, and instance types are available now and you can start using them today!

— Jeff;

Amazon Lightsail makes it easy for you to get started with AWS. You choose the operating system (and optional application) that you want to run, pick an instance plan, and create an instance, all in a matter of minutes. Lightsail offers low, predictable pricing, with instance plans that include compute power, storage, and data transfer:

Managed Databases
Today we are making Lightsail even more useful by giving you the ability to create a managed database with a couple of clicks. This has been one of our top customer requests and I am happy to be able to share this news.

This feature is going to be of interest to a very wide range of current and future Lightsail users, including students, independent developers, entrepreneurs, and IT managers. We’ve addressed the most common and complex issues that arise when setting up and running a database. As you will soon see, we have simplified and fine-tuned the process of choosing, launching, securing, accessing, monitoring, and maintaining a database!

Each Lightsail database bundle has a fixed, monthly price that includes the database instance, a generous amount of SSD-backed storage, a terabyte or more of data transfer to the Internet and other AWS regions, and automatic backups that give you point-in-time recovery for a 7-day period. You can also create manual database snapshots that are billed separately.

Creating a Managed Database
Let’s walk through the process of creating a managed database and loading an existing MySQL backup into it. I log in to the Lightsail Console and click Databases to get started. Then I click Create database to move forward:

I can see and edit all of the options at a glance. I choose a location, a database engine and version, and a plan, enter a name, and click Create database (all of these options have good defaults; a single click often suffices):

We are launching with support for MySQL 5.6 and 5.7, and will add support for PostgreSQL 9.6 and 10 very soon. The Standard database plan creates a database in one Availability Zone with no redundancy; the High Availability plan also creates a presence in a second AZ, and is recommended for production use.

Database creation takes just a few minutes, the status turns to Available, and my database is ready to use:

I click on Database-Oregon-1, and I can see the connection details, and have access to other management information & tools:

I’m ready to connect! I create an SSH connection to my Lightsail instance, ensure that the mysql package is installed, and connect using the information above (read Connecting to Your MySQL Database to learn more):

Now I want to import some existing data into my database. Lightsail lets me enable Data import mode in order to defer any backup or maintenance operations:

Enabling data import mode deletes any existing automatic snapshots; you may want to take a manual snapshot before starting your import if you are importing fresh data into an existing database.

I have a large (13 GB) , ancient (2013-era) MySQL backup from a long-dead personal project; I download it from S3, uncompress it, and import it:

I can watch the metrics while the import is underway:

After the import is complete I disable data import mode, and I can run queries against my tables:

To learn more, read Importing Data into Your Database.

Lightsail manages all routine database operations. If I make a mistake and mess up my data, I can use the Emergency Restore to create a fresh database instance from an earlier point in time:

I can rewind by up to 7 days, limited to when I last disabled data import mode.

I can also take snapshots, and use them later to create a fresh database instance:

Things to Know
Here are a couple of things to keep in mind when you use this new feature:

Engine Versions – We plan to support the two latest versions of MySQL, and will do the same for other database engines as we make them available.

High Availability – As is always the case for production AWS systems, you should use the High Availability option in order to maintain a database footprint that spans two Zones. You can switch between Standard and High Availability using snapshots.

Scaling Storage – You can scale to a larger database instance by creating and then restoring a snapshot.

Data Transfer – Data transfer to and from Lightsail instances in the same AWS Region does not count against the usage that is included in your plan.

Amazon RDS – This feature shares core technology with Amazon RDS, and benefits from our operational experience with that family of services.

Available Now
Managed databases are available today in all AWS Regions where Lightsail is available:

— Jeff;

As I write this, there are just 55 calendar days until AWS re:Invent 2018. My colleagues and I are working flat-out to bring you the best possible learning experience and I want to give you a quick update on a couple of things…

Transportation – Customer Obsession is the first Amazon Leadership Principle and we take your feedback seriously! The re:Invent 2018 campus is even bigger this year, and our transportation system has been tuned and scaled to match. This includes direct shuttle routes from venue to venue so that you don’t spend time waiting at other venues, access to real-time transportation info from within the re:Invent app, and on-site signage. The mobile app will even help you to navigate to your sessions while letting you know if you are on time. If you are feeling more independent and don’t want to ride the shuttles, we’ll have partnerships with ridesharing companies including Lyft and Uber. Visit the re:Invent Transportation page to learn more about our transportation plans, routes, and options.

Reserved Seating – In order to give you as many opportunities to see the technical content that matters the most to you, we are bringing back reserved seating. You will be able to make reservations starting at 10 AM PT on Thursday, October 11, so mark your calendars. Reserving a seat is the best way to ensure that you will get a seat in your favorite session without waiting in a long line, so be sure to arrive at least 10 minutes before the scheduled start. As I have mentioned before, we have already scheduled repeats of the most popular sessions, and made them available for reservation in the Session Catalog. Repeats will take place all week in all re:Invent venues, along with overflow sessions in our Content Hubs (centralized overflow rooms in every venue). We will also stream live content to the Content Hubs as the sessions fill up.

Trivia Night – Please join me at 7:30 PM on Wednesday in the Venetian Theatre for the first-ever Camp re:Invent Trivia Night. Come and test your re:Invent and AWS knowledge to see if you and your team can beat me at trivia (that should not be too difficult). The last person standing gets bragging rights and an awesome prize.

How to re:Invent – Whether you are a first-time attendee or a veteran re:Invent attendee, please take the time to watch our How to re:Invent videos. We want to make sure that you arrive fully prepared, ready to learn about the latest and greatest AWS services, meet your peers and members of the AWS teams, and to walk away with the knowledge and the skills that will help you to succeed in your career.

See you in Vegas!

— Jeff;

AWS Online Tech Talks are live, online presentations that cover a broad range of topics at varying technical levels. Join us this month to learn about AWS services and solutions. We’ll have experts online to help answer any questions you may have.

Featured this month, check out the webinars under AR/VR, End-User Computing, Industry Solutions. Also, register for our second fireside chat discussion on Amazon Redshift.

Register today!

Note – All sessions are free and in Pacific Time.

Tech talks featured this month:

AR/VR

October 16, 2018 | 01:00 PM – 02:00 PM PT – Creating and Publishing AR, VR and 3D Applications with Amazon Sumerian – Learn about Amazon Sumerian, the fastest and easiest way to create and publish immersive applications.

Compute

October 25, 2018 | 09:00 AM – 10:00 AM PT – Running Cost Effective Batch Workloads with AWS Batch and Amazon EC2 Spot Instances – Learn how to run complex workloads, such as analytics, image processing, and machine learning applications efficiently and cost-effectively.

Data Lakes & Analytics

October 18, 2018 | 01:00 PM – 01:45 PM PT – Fireside Chat: The Evolution of Amazon Redshift – Join Vidhya Srinivasan, General Manager of Redshift, in a candid conversation as she discusses the product’s evolution recently shipped features and improvements.

October 15, 2018 | 01:00 PM – 01:45 PM PT – Customer Showcase: The Secret Sauce Behind GroupM’s Marketing Analytics Platform – Learn how GroupM – the world’s largest media investment group with more than $113.8bn in billings – created a modern data analytics platform using Amazon Redshift and Matillion.

Databases

October 15, 2018 | 11:00 AM – 12:00 PM PT – Supercharge Query Caching with AWS Database Services – Learn how AWS database services, including Amazon Relational Database Service (RDS) and Amazon ElastiCache, work together to make it simpler to add a caching layer to your database, delivering high availability and performance for query-intensive apps.

October 17, 2018 | 09:00 AM – 09:45 AM PT – How to Migrate from Cassandra to DynamoDB Using the New Cassandra Connector in the AWS Database Migration Service – Learn how to migrate from Cassandra to DynamoDB using the new Cassandra Connector in the AWS Database Migration Service.

End-User Computing

October 23, 2018 | 01:00 PM – 02:00 PM PT – How to use Amazon Linux WorkSpaces for Agile Development – Learn how to integrate your Amazon Linux WorkSpaces development environment with other AWS Developer Tools.

Enterprise & Hybrid

October 23, 2018 | 09:00 AM – 10:00 AM PT – Migrating Microsoft SQL Server 2008 Databases to AWS – Learn how you can provision, monitor, and manage Microsoft SQL Server on AWS.

Industry Solutions

October 24, 2018 | 11:00 AM – 12:00 PM PT – Tape-to-Cloud Media Migration Walkthrough – Learn from media-specialist SAs as they walk through a content migration solution featuring machine learning and media services to automate processing, packaging, and metadata extraction.

IoT

October 22, 2018 | 01:00 PM – 01:45 PM PT – Using Asset Monitoring in Industrial IoT Applications – Learn how AWS IoT is used in industrial applications to understand asset health and performance.

Machine Learning

October 15, 2018 | 09:00 AM – 09:45 AM PT – Build Intelligent Applications with Machine Learning on AWS – Learn how to accelerate development of AI applications using machine learning on AWS.

Management Tools

October 24, 2018 | 01:00 PM – 02:00 PM PT – Implementing Governance and Compliance in a Multi-Account, Multi-Region Scenario – Learn AWS Config best practices on how to implement governance and compliance in a multi-account, multi-Region scenario.

Networking

October 23, 2018 | 11:00 AM – 11:45 AM PT – How to Build Intelligent Web Applications @ Edge – Explore how Lambda@Edge can help you deliver low latency web applications.

October 25, 2018 | 01:00 PM – 02:00 PM PT – Deep Dive on Bring Your Own IP – Learn how to easily migrate legacy applications that use IP addresses with Bring Your Own IP.

re:Invent

October 10, 2018 | 08:00 AM – 08:30 AM PT – Episode 6: Mobile App & Reserved Seating – Discover new innovations coming to the re:Invent 2018 mobile experience this year. Plus, learn all about reserved seating for your priority sessions.

Security, Identity & Compliance

October 22, 2018 | 11:00 AM – 11:45 AM PT – Getting to Know AWS Secrets Manager – Learn how to protect your secrets used to access your applications, services, and IT resources.

Serverless

October 17, 2018 | 11:00 AM – 12:00 PM PT – Build Enterprise-Grade Serverless Apps – Learn how developers can design, develop, deliver, and monitor cloud applications as they take advantage of the AWS serverless platform and developer toolset.

Storage

October 24, 2018 | 09:00 AM – 09:45 AM PT – Deep Dive: New AWS Storage Gateway Hardware Appliance – Learn how you can use the AWS Storage Gateway hardware appliance to connect on-premises applications to AWS storage.

Today is Save the Koala Day and a perfect time to to tell you about some noteworthy and ground-breaking research that was made possible by AWS Research Credits and the AWS Cloud.

Five years ago, a research team led by Dr. Rebecca Johnson (Director of the Australian Museum Research Institute) set out to learn more about koala populations, genetics, and diseases. As a biologically unique animal with a limited appetite, maintaining a healthy and genetically diverse population are both key elements of any conservation plan. In addition to characterizing the genetic diversity of koala populations, the team wanted to strengthen Australia’s ability to lead large-scale genome sequencing projects.

Inside the Koala Genome
Last month the team published their results in Nature Genetics. Their paper (Adaptation and Conservation Insights from the Koala Genome) identifies the genomic basis for the koala’s unique biology. Even though I had to look up dozens of concepts as I read the paper, I was able to come away with a decent understanding of what they found. Here’s my lay summary:

Toxic Diet – The eucalyptus leaves favored by koalas contain a myriad of substances that are toxic to other species if ingested. Gene expansions and selection events in genes encoding enzymes with detoxification functions enable koalas to rapidly detoxify these substances, making them able to subsist on a diet favored by no other animal. The genetic repertoire underlying the accelerated metabolics also renders common anti-inflammatory medications and antibiotics ineffective for treating ailing koalas.

Food Choice – Koalas are, as I noted earlier, very picky eaters. Genetically speaking, this comes about because their senses of smell and taste are enhanced, with 6 genes giving them the ability to discriminate between plant metabolites on the basis of smell. The researchers also found that koalas have a gene that helps them to select eucalyptus leaves with a high water content, and another that enhances their ability to perceive bitter and umami flavors.

Reproduction – Specific genes which control ovulation and birth were identified. In the interest of frugality, female koalas produce eggs only when needed.

Koala Milk – Newborn koalas are the size of a kidney bean and weigh less than half of a gram! They nurse for about a year, taking milk that changes in composition over time, with a potential genetic correlation. The researchers also identified genes known to function as anti-microbial properties.

Immune Systems – The researchers identified genes that formed the basis for resistance, immunity, or susceptibility to certain diseases that affect koalas. They also found evidence of a “genomic invasion” (their words) where the koala retrovirus actually inserts itself into the genome.

Genetic Diversity – The researchers also examined how geological events like habitat barriers and surface temperatures have shaped genetic diversity and population evolution. They found that koalas from some areas had markedly less genetic diversity than those from others, with evidence that allowed them to correlate diversity (or the lack of it) with natural barriers such as the Hunter Valley.

Powered by AWS
Creating a complete gene sequence requires (among many other things) an incredible amount of compute power and vast amount of storage.

While I don’t fully understand the process, I do know that it works on a bottom-up basis. The DNA samples are broken up into manageable pieces, each one containing several tens of thousands of base pairs. A variety of chemicals are applied to cause the different base constituents (A, T, C, or G) to fluoresce, and the resulting emission is captured, measured, and stored. Since this study generated a koala reference genome, the sequencing reads were assembled using an overlapping layout consensus assembly algorithm known as Falcon which was run on AWS. The koala genome comes in at 3.42 billion base pairs, slightly larger than the human genome.

I’m happy to report that this groundbreaking work was performed on AWS. The research team used cfnCluster to create multiple clusters, each with 500 to 1000 vCPUs, and running Falcon from Pacific Biosciences. All in all, the team used 3 million EC2 core hours, most of which were EC2 Spot Instances. Having access to flexible, low-cost compute power allowed the bioinformatics team to experiment with the configuration of the Falcon pipeline as they tuned and adapted it to their workload.

We are happy to have done our small part to help with this interesting and valuable research!

— Jeff;

The Altair 8800 computer that I built in 1977 had just 4 kilobytes of memory. Today I was able to use an EC2 instance with 12 terabytes (12 tebibytes to be exact) of memory, almost 4 billion times as much!

The new Amazon EC2 High Memory Instances let you take advantage of other AWS services including Amazon Elastic Block Store (EBS), Amazon Simple Storage Service (S3), AWS Identity and Access Management (IAM), Amazon CloudWatch, and AWS Config. They are designed to allow AWS customers to run large-scale SAP HANA installations, and can be used to build production systems that provide enterprise-grade data protection and business continuity.

Here are the specs:

Each Logical Processor is a hyperthread on one of the 224 physical CPU cores. All three sizes are powered by the latest generation Intel^® Xeon^® Platinum 8176M (Skylake) processors running at 2.1 GHz (with Turbo Boost to 3.80 GHz), and are available as EC2 Dedicated Hosts for launch within a new or existing Amazon Virtual Private Cloud (VPC). You can launch them using the AWS Command Line Interface (CLI) or the EC2 API, and manage them there or in the EC2 Console.

The instances are EBS-Optimized by default, and give you low-latency access to encrypted and unencrypted EBS volumes. You can choose between Provisioned IOPS, General Purpose (SSD), and Streaming Magnetic volumes, and can attach multiple volumes, each with a distinct type and size, to each instance.

SAP HANA in Minutes
The EC2 High Memory instances are certified by SAP for OLTP and OLAP workloads such as S4/HANA, Suite on HANA, BW4/HANA, BW on HANA, and Datamart (see the SAP HANA Hardware Directory for more information).

We ran the SAP Standard Application Benchmark and measured the instances at 480,600 SAPS, making them suitable for very large workloads. Here’s an excerpt from the benchmark:

In anticipation of today’s launch, the EC2 team provisioned a u-12tb1.metal instance for my AWS account and I located it in the Dedicated Hosts section of the EC2 Console:

Following the directions in the SAP HANA on AWS Quick Start, I copy the Host Reservation ID, hop over to the CloudFormation Console and click Create Stack to get started. I choose my template, give my stack a name, and enter all of the necessary parameters, including the ID that I copied, and click Next to proceed:

On the next page I indicate that I want to tag my resources, leave everything else as-is, and click Next:

I review my settings, acknowledge that the stack might create IAM resources, and click Next to create the stack:

The AWS resources are created and SAP HANA is installed, all in less than 40 minutes:

Using an EC2 instance on the public subnet of my VPC, I can access the new instance. Here’s the memory:

And here’s the CPU info:

I can also run an hdbsql query:

SELECT 
  DISTINCT HOST, CAST(VALUE/1024/1024/1024 AS INTEGER) AS TOTAL_MEMORY_GB 
  FROM SYS.M_MEMORY
  WHERE NAME='SYSTEM_MEMORY_SIZE';

Here’s the output, showing that SAP HANA has access to 12 TiB of memory:

Another option is to have the template create a second EC2 instance, this one running Windows on a public subnet, and accessible via RDP:

I could install HANA Studio on this instance and use its visual interface to run my SAP HANA queries.

The Quick Start implementation uses high performance SSD-based EBS storage volumes for all of your data. This gives you the power to switch to a larger instance in minutes without having to migrate any data.

Available Now
Just like the existing SAP-certified X1 and X1e instances, the EC2 High Memory instances are very cost-effective. For example, the effective hourly rate for the All Upfront 3-Year Reservation for a u-12tb1.metal Dedicated Host in the US East (N. Virginia) Region is $30.539 per hour.

These instances are now available in the US East (N. Virginia) and Asia Pacific (Tokyo) Regions as Dedicated Hosts with a 3-year term, and will be available soon in the US West (Oregon), Europe (Ireland), and AWS GovCloud (US) Regions. If you are ready to get started, contact your AWS account team or use the Contact Us page to make a request.

In the Works
We’re not stopping at 12 TiB, and are planning to launch instances with 18 TiB and 24 TiB of memory in 2019.

— Jeff;

PS – If you have applications that might need multiple terabytes in the future but can run comfortably in less memory today, be sure to consider the R5, X1, and X1e instances.