Management of Security Group ingress rules to allow access to dynamic IPs.

Our infrastructures are designed to remain isolated in a private subnet of the VPC, this in order to increase the security level and block any type of unwanted access directly from network layer. For various management reasons like verification and troubleshooting we need to access the resources inside these isolated infrastructures.

The common solution for this type of setup is the 

, a world-exposed EC2 instance that allows access to other isolated resources. To block any unwanted access we use a whitelist system through the ingress rules of the security group of the bastion instance. Any SSH access is blocked except for certain IPs which are manually added when requested.

This system is very robust: it delegates access control to the security group, a resource managed by AWS, by doing so we do not have to take on the charge of system configurations (for example iptables) and manage various and possible security patches.

This solution is great when using static IPs, a request is made to the DevOps team to whitelist a certain IP and this is added among the ingress rules for the specific port of service requested (port 22 for SSH access). This works well when the access request comes from other infrastructures or offices or coworking that have a static IP.

Since our company is distributed and adopting the 

, developers or DevOps team members often connect from networks with dynamic IPs. This means we have to continuously manage security group rules of the bastion instance by removing and adding IPs almost every day.

After evaluating some solutions, we decided to adopt a serverless system based on API Gateway and Lambda that automatically adds the IP to whitelist on the security group. To dig more into detail: we have exposed an HTTP API using AWS API Gateway service that, upon an HTTP request id received, it trigger the execution of a Lambda. The Lambda function retrieves the IP of the caller from the input event and modifies security group ingress rules to allow access for SSH service.

This system is very simple and being based on serverless services, with very low traffic, the cost is zero. The Lambda code is also very simple:

As you can see, to verify the origin of the request we opted a solution that pass a certain token as a path parameter. This is because we want the use of this endpoint to be as simple as possible and in this way we can directly open a link similar to this:

https://xxxxxxxxx.execute-api.us-east-1.amazonaws.com/stage/whitelist/yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

directly in the browser without having to set a particular header (for example 

 using the API Gateway build-in system) that would need, for example, a curl call from the terminal.

The configuration of this token, security group ID, protocol, port and description is done through environment variables, in this way we can deploy the same Lambda code for different environments and infrastructures.

 (incrementable up to 1000 via a request to AWS) which may be affected after a few months of system usage. In addition, this automatic system is used exclusively by those who connect using a dynamic IP, this means that the IP will be reassigned to someone else and must therefore be removed from whitelist.

The solution we have adopted is to add another Lambda, this time triggered by a scheduled CloudWatch event, which will clean up the security group rules.

Obviously we do not want the rules added manually for static IPs to be removed as well, otherwise it would cause an inadvertent blocking of third-party integration accesses. To solve this problem we use as discriminant the rule's description, set through the environment variable 

 in the Lambda described above. In this way we are going to clean up only the rules added through this automatic system and allowing the DevOps team to manage some rules manually.

The Lambda code is also very simple in this case:

 API by passing the list of ingress rules with a specific description.

This system meets our needs and has a good compromise between security and usability. This system only grants access by a certain IP to the SSH service exposed by the bastion but does not grant access to the machine, whoever wants to connect must still use their own personal SSH key.

Ingress rules for dynamic IPs

Regole di ingresso per IP dinamici

Negli ultimi anni personalmente non ho mai perso un’edizione di quello che considero 

il più importante evento in Italia dedicato all’e-commerce

. Vuoi perché è interessante vedere come evolve l’offerta di servizi che ruotano attorno a questo mondo in rapida evoluzione, vuoi perché è stato sempre un’ottima occasione per concentrare in un paio di giorni una serie di incontri con quei partner, prospect e clienti con i quali, per via della distanza territoriale, capita raramente di vedersi di persona.

Mai prima di oggi, però, avevamo preso in considerazione di partecipare come espositori per un motivo molto semplice: il servizio da noi erogato era principalmente legato allo 

sviluppo tecnologico sulla piattaforma Magento

 e, come potete immaginare, la richiesta portata dal solo passaparola è sempre stata ben superiore all’offerta che potevamo mettere a disposizione con un team in crescita costante, ma organica al fine di preservare il livello di competenza e qualità. In una situazione come questa farsi conoscere da un pubblico più ampio rischia di essere più controproducente che altro. Qualche tempo fa avevo scritto anche 

Prima per offrire un servizio di qualità ai nostri clienti, poi come effettiva opportunità di business, nel corso degli ultimi anni in Bitbull abbiamo iniziato a maturare competenze sempre più verticali sulle infrastrutture cloud e, in particolare, sfruttando i servizi forniti da Amazon Web Services. Questo tipo di soluzione si è dimostrata ideale per coprire le esigenze di scalabilità, affidabilità e distribuzione territoriale di chi con l’e-commerce si trova a dover sostenere numeri sempre più importanti sia per quanto riguarda il traffico sia per quanto riguarda l’acquisizione di ordini.

Un percorso, il  nostro, che ha avuto  una naturale evoluzione l’anno scorso, il 2018, anno in cui siamo diventati ufficialmente partner. Questo  passo, di fatto, ci ha consentito di aprire un dialogo diretto con AWS e ci ha permesso di cogliere l’opportunità di presentarci insieme all’edizione 2019 del Netcomm Forum.

Lo scopo principale della nostra partecipazione è quello di presentare un servizio per il quale ci siamo resi conto non esserci molta concorrenza in Italia, il cui valore risiede principalmente nell’unire le 

 su una determinata piattaforma e-commerce (ad oggi Magento, ma stiamo ampliando l’offerta ad altre soluzioni) alle 

 verticali su uno strumento leader mondiale nei servizi in cloud. Come spesso succede, quando ci si affaccia al mercato lavorando bene in un determinato ambito, le opportunità si allargano anche ad altri settori, motivo per cui oggi ci ritroviamo a gestire infrastrutture che vanno anche oltre al mondo dell’e-commerce.

Come potrà notare chi passerà a farci visita al nostro stand, puntiamo molto sul tema dell’

. Pur avendo un forte background tecnologico, ci siamo resi conto in questi anni di quanto la tecnologia, a seconda dell’utilizzo che se ne fa, possa incidere tanto positivamente quanto negativamente in un rapporto costi / benefici nel 

Un uso intelligente e pragmatico degli strumenti che oggi abbiamo a disposizione può portare enormi benefici non soltanto al processo di vendita attraverso il canale digitale, ma anche nell’ottimizzazione di tutta una serie di processi ad esso correlati che vanno dalla logistica all’amministrazione, dalla gestione del proprio catalogo fino al customer care. Una visione a 360° permette quindi di portare l’investimento tecnologico ad essere funzionale sia ad una riduzione dei costi di gestione, sia ad un’ottimizzazione dei processi che impattano su tutto l’ambito aziendale.

Questa consapevolezza ci ha fatto capire che il valore che possiamo portare va oltre a quello del “semplice” partner tecnologico, abbiamo quindi iniziato a proporre un 

 che  sta portando riscontri particolarmente positivi da parte dei nostri clienti, ormai costretti a doversi confrontare con un mondo in rapida evoluzione che impone un ripensamento di attività consolidate da anni, ma talvolta incompatibili con le attuali esigenze del mercato.

Quando decidiamo di fare un passo in genere preferiamo non limitarci al semplice compitino, quindi abbiamo deciso di tenere anche un workshop per raccontare tramite un caso reale come, utilizzando i giusti strumenti, è possibile migliorare le prestazioni ed ottenere una maggiore stabilità ottimizzando contestualmente i costi. Il tutto senza cambiare il fornitore dei servizi in cloud, ma sfruttando al meglio le caratteristiche e le opportunità che esso offre.

Per maggiori informazioni vi rimando alla 

, la speranza ovviamente sarà quella di potervi accogliere numerosi! :)

Sul perché quest’anno saremo presenti al Netcomm e perché lo saremo insieme ad Amazon Web Services

? I've heard about it so many, too many maybe,so I decided to embark on this journey to discover this new technology.

So I prepare the backpack, the boots, rope, harness and I start..

Outside is a gloomy, autumn day, the right climate to stay in houses with a cup of coffee and my notebook. Click on my faithful companion Firefox, where to start? First of all I have to choose which cloud provider to use ..5 seconds to think.. AWS of course.

The first research leads me to hundreds and hundreds of blog articles, really so much material. I try to make a summary so I convince myself to have understood:

Do you know servers? Those strange computers, in dark rooms, ok, out of fashion anymore... why? They are difficult to manage, you have to use a thousand precautions, uninterruptible power supply in case the light goes out, the right temperature, absolutely no dust, fire doors, fire systems with argon etc..etc.. so to ensure a stable service without become crazy and spend a lot of money services like AWS, Google Cloud, Azure and many others "rent" part of their data center, you only have to configure the infrastructure with a few clicks and only worry about the software part.

After this great step forward there are other problems (and of course nobody likes them), you have to keep up-to-date the operating system installed on your virtual servers, install security patches, configure a scaling of the computational resources, redundate the infrastructure in order to guarantee an high reliability of the service.. and all that, so what have the cloud providers invented to come up against? Manage some of the services like MySQL, Elasticsearch, Redis, RabbitMQ and many others, directly accessing it without having to take care of installing and updating these services, obviously the configuration (even if a little limited) is always possible.

After this last step still remains the problem of virtual machines that will run your software (always the same problem mentioned above) so what is the next step? The Serverless! That is, managed platforms where the only thing you need to worry about is your code, nothing else, it's the cloud provider that will automatically take care of scaling, updating the operating systems that host your code and ensuring its reliability.

Server-less does not mean that there are no servers but that you do not need to manage them :)

After the initial study done yesterday, today is the day of the practical test! There are many guides that explain how to create a simple API using AWS services, but the activation steps of the services are done directly from the web console (or using CLI), however convenient it does not allow you to easily reproduce the configuration steps and the changes are not traced. AWS provides a service called 

, by describing the infrastructure with a JSON or YAML file you can provision all the resources you need in one go, manage updates and rollback if an error occurs.

Of course, writing a huge JSON or YAML file is not very convenient, so I looked for what AWS framework provides:

Honestly I found a bit of confusion about what to use but the advice is: SAM for API services, Mobile CLI to avoid, Amplify for more complex applications also equipped with frontend. I then focused on the latter, to see the full experience and see the news.

Very very easy to use, maybe even too much, just download the cli

enter the root folder of your web application and add Amplify as an addiction

and as a last part initialize the CloudFormation base stack where just a couple of access rules will be created and little else

Now the fun part comes up, add services that our application needs, the Amplify CLI will always use guided steps to configure them. By making an overview of what we can add, we see:

Wow! There is practically everything and you can add one command

and once the selected service has been set up, deploy the changes

Really nice, some services I already knew them as 

Amplify is really a great tool, especially for the integrations for React, AngularJS and Vue.js that allow you to connect services quickly using the framework that we like best. It took 15 minutes to get a simple application (yes, the usual old TODO list) up&running, but if you do not get your hands dirty by putting them in the engine, I do not feel completely satisfied.

I then start to integrate my application written in 

 starting from authentication. The configuration of Cognito is really simplified:

1 minute and everything is ready to manage the users.

I am amazed at how much functionality Cognito offers, confirmation of the email and/or telephone number sending you a code (via SMS for the phone), login with Google, Facebook, Amazon and OpenID, and provide a complete self-managed OAuth2 server. To create this, from scratch, with a PHP server or nodeJS at least 1/2 days of work would be served, for this I spent 1 minute, it's crazy.

Now the database, I had already used in the past DynamoDB for some additions but never in combination with 

, a fairly recent service (no more than 1 year and a half) that helps you manage access to your data using GraphQL endpoints. It is appropriate to say: the latest fashion!

With Amplify it was really simple, you certainly need to have some experience with GraphQL.

? Please select from one of the below mentioned services (Use arrow keys)
❯ GraphQL
 REST

not only GraphQL endpoints but also REST! a step forward to choose the authentication method

? Please select from one of the below mentioned services GraphQL
? Provide API name: myNotesApi
? Choose an authorization type for the API (Use arrow keys)
❯ API key
 Amazon Cognito User Pool

you can use either static keys (generated during this process) or using the authentication system created first with Cognito, obviously I choose the last option.

Today I started writing models for my application, one of Amplify's most convenient features is the automatic generation of GraphQL resolvers, even with 

 to specify the access level. This last part is vital to restrict user access only to their data and not to the entire DynamoDB table.

Here the first problem, I tried to faithfully follow the documentation but nothing, checking the resolvers generated I saw no difference using or not the transformer 

. I lost a lot of time on this until, I gave up, I created an 

 on the Github respository to ask for support, they really answered me rapidly and it turned out that actually it was not really working... okay, things that can happen.

During debugging I discovered that the DynamoDB table indexes were created with the 

 for Reads and Writes to 5, without possibility of being modified.

For those who do not know, this value is one of the parameters that is used to calculate the cost of the service of DynamoDB: the number of accesses in reading or writing per second (not so simple but I do not enter into the specification). In order not to go into too much detail, this non-relational database answers for each query at the maximum possible speed regardless of the amount of data and there is no limit to the number of lines (pay for the weight in GB of the table). The negative part is a somewhat tricky query engine and you need to carefully create table indexes that are vital for sorting and querying.

I did not expect that for any test you will pay at least around $13, a bit expensive considering that with less than $1 I have 2 projects of IoT, a GoogleHome action, my website (with two environments) and I use S3 to make an encrypted backup of some important files. With just 5/6 models I arrived at disproportionate figures, with a DynamoDB table ready for production when the application was not born yet. I then opened another 

, this time not yet taken into consideration for now.

So I eliminated the project and I got lost, I had found a very easy to use and powerful tool but it's still too young to be used for production. Being so simple it does not even allow such a deep changes of some parts of it except with different workarounds.

Several nights are spent evaluating alternatives.

Over the days I have collected several alternatives to Amplify, difficult because there are no other CLI that with a couple of commands do EVERYTHING.

, a very powerful and modular framework with a large community behind which develops hundreds of plugins. I've already used it before for some cloud infrastructure and API automation projects and I've always been comfortable with it.

The management of the resources directly uses the CloudFormation templates, this guarantees to be always updated and uses the functionalities served by AWS without having to report the changes within the framework (an example, 

 uses its syntax and its integration, can not keep up and I always found dozens of bugs).

When there are many resources to add out of those managed by Serverless the 

 explains how to create them in the file, not make the mistake of putting everything in a single file of 2000 lines that is difficult to maintain, better divide it into multiple files

resources:
 - ${file(resources/cognito.yml)}
 - ${file(resources/userRole.yml)}
 - ${file(resources/queues.yml)}
 - ${file(resources/tables/todos.yml)}
 - ${file(resources/tables/projects.yml)}
 - ${file(resources/tables/categories.yml)}
 - ${file(resources/outputs.yml)}

and then in these files specify the resources

Resources:
 AccountTable:
   Type: AWS::DynamoDB::Table
   Properties:
     TableName: "${self:service}-${self:provider.stage}-${self:custom.todosTableName}"
     AttributeDefinitions:
       -
         AttributeName: userId
         AttributeType: S
       -
         AttributeName: id
         AttributeType: S
     KeySchema:
       -
         AttributeName: userId
         KeyType: HASH
       -
         AttributeName: id
         KeyType: RANGE
     ProvisionedThroughput:
       ReadCapacityUnits: 1
       WriteCapacityUnits: 1
     StreamSpecification:
       StreamViewType: "NEW_AND_OLD_IMAGES"

in this way the code will be much more organized and the files smaller and more maintainable.

The previous speech also applies to the configurations of the project, better specify a separate file so you can change it depending on the environment or project without using hundreds of environment variables

service: MyService
custom: ${file(./config.yml)}

provider:
 name: aws
 runtime: nodejs8.10
 stage: ${opt:stage,self:custom.env}
 region: ${self:custom.region}

the configuration file becomes much easier to view

env: "Stage"
region: "eu-west-1"

todosTableName: "Todos"

Attracted by Amplify to use GraphQL I immediately started using 

, easy to use and, with some organization and re-use of resolvers, I arrived at a good point. I realized that I was writing so much code that meant keeping it, testing it, optimizing it, and taking a lot of time to create a simple CRUD feature for the models.

A doubt has arisen: do I really need a GraphQL APIs? Of course they are really comfortable, expressive and easy to keep up to date but why? More specifically: do I need APIs to consume? The APIs are mainly used to access the database, add server-side logic (which the user can not manipulate) and make the service available to third-party services.

DynamoDB has its own build-in functionality to limit access to rows, columns and operations based on the user, ie a user can only see certain attributes of the saved object (the "columns"), access only the objects that he has created (the "lines") and read some tables without the possibility of modifying it this by identifying the user through Cognito authentication.
What does it mean? Which has a security system for CRUD access already served, without having to make these checks via code, it is already done! It's AWS to guarantee this security, do you want to be a developer to do better than the team of 100 Amazon developers? You could, but in 99% of cases you're missing some vulnerabilities.

You can say everything about the AWS ecosystem according to the opinions, that is inconvenient, expensive, difficult, complicated but it can not be said that it's not safe, so ok, I give them this part.

Side note: once authenticated a user with Cognito can access any AWS resource limiting access and operations through the 

, not all resources support a granularity like DynamoDB but already only for file upload the S3 service can do it (think about the user's avatar, for example, convenient!).

I need it? What am I doing that I have to hide from the user? If I already know that he will only access his lines at most is breaking the formatting of his data, well, he will not be able to view them correctly.

Surely soon or later there will be the need to call third party services to add some functionality, no problem, with 

 where you can do whatever you want, the events that can trigger this custom logic are endless: from creating or editing an object in DynamoDB to uploading a file to S3, even all the flow of authentication/registration/confirmation of Cognito users, no more.

Do you need it already? Do we already have partners who ask to be able to take advantage of our services? if the answer is yes, you can create the entire API compartment you want with Serverless 

 creating only the endpoints you need, otherwise you can skip this part.

 is an AWS service that allows you to organize API endpoints and manipulate request and response parameters.

No, I do not need them for the moment. I wanted to give Amplify another chance and use the client directly to access DynamoDB and use the AWS resources I need. Very clean and really simple to use.

In any case, for authentication to the API endpoint you can use either Cognito directly (useful in case calls from the same application that you need to login) of the fixed keys and an access key generated 

from the OAuth2 endpoint served by Cognito

. This last option makes the endpoint very professional by using the security of an OAuth2 access and allows configuring custom scopes to regulate access to resources.

Once the backend part was put in place, it was time to start creating the frontend part. I did not have much doubt about the technology to use, having enough experience with 

and following the wizard I chose a very simple configuration and then saved as a preset. The configuration to insert in the 

{
 "packageManager": "npm",
 "useTaobaoRegistry": false,
 "presets": {
   "vue-router-vuex": {
     "useConfigFiles": true,
     "plugins": {
       "@vue/cli-plugin-babel": {},
       "@vue/cli-plugin-pwa": {},
       "@vue/cli-plugin-eslint": {
         "config": "base",
         "lintOn": [
           "commit"
         ]
       }
     },
     "router": true,
     "routerHistoryMode": true,
     "vuex": true,
     "cssPreprocessor": "stylus"
   },
 }
}

? It was a choice driven by the component library I want to use: 

. I've already used it for some projects and I've always felt good, I find it very complete for the quantity of components and, very important thing: very customizable.

Here too, thanks to the CLI, a command was enough to add this library along with some sample configurations

My serverless journey

Il mio viaggio nel mondo serverless

Temporized RDS read replica AutoScaling with Lambda and Route53

One of our hosting infrastructure on AWS recently has encountered some problems due to a huge and rapid increase in traffic during some pre-defined periods, at the same time as a newsletter, that quickly brought many more users to visit our customer’s Magento e-commerce.

The number of requests made to our ElasticLoadBalancer ranged from 400 to 3,000 per minute in less than 5 minutes, that did not give us enough time to scale-up the infrastructure before it fall down.
From our analysis we have identified the bottleneck in the database, especially in categories and catalog read queries, dynamics based on various configurations and based on the connected user.

To better control the load on the database, we have decided to use two RDS instances, a master for write operations, and a replica for read operations to only scale the necessary part and control costs.
We noticed an improvement but the same reading replication suffered from the load and they quickly saturated the database connections.

To prepare us for incoming traffic, one hour before the peak of traffic, we changed the database endpoint from the Magento configurations to not use the replication. When the scaling was done we brought the configuration back to the previous one.

This solution worked fine but it was very uncomfortable to change the Magento configurations, so we had to empty the cache and this would be a further burden on the first few site visits in addition to having to manually process the procedures with the risk of human error.

The solution we adopted for not having to modify the connection configurations of the Magento database endpoint was to use internal VPC DNS managed with Route53 by mapping the hosts of RDS instances with domain names using CNAME records in the magento configurations file.

This way we have the control of pointing the web application to the database agnostically to its configurations. This allows us to redirect the read replication connections to the master during scaling and this not generate any downtime.

Once we have reached a stable solution that allows us to scale without downtime, it only remains to set this procedure automatically.

"This is a task for Lambda," we said, so we wrote two Lambda functions in NodeJS that are able to modify the RDS instance and the Route53 DNS record attached to its endpoint.
For this small project we chose to use the Serverless framework that we used in the past with great satisfaction.

functions:
  routingToMaster:
    handler: route53.changeRecord
    environment:
      Msg: "${self:custom.MsgRoutingToMaster}"
      HostedZoneId: "${self:custom.HostedZoneId}"
      RecordName: "${self:custom.RecordName}"
      RecordValue: "${self:custom.MasterEndpoint}"

  scaleUp:
    handler: rds.scale
    environment:
      Msg: "${self:custom.MsgScaleUp}"
      InstanceIdentifier: "${self:custom.InstanceIdentifier}"
      InstanceClass: "${self:custom.InstanceClassBig}"

  scaleDown:
    handler: rds.scale
    environment:
      Msg: "${self:custom.MsgScaleDown}"
      InstanceIdentifier: "${self:custom.InstanceIdentifier}"
      InstanceClass: "${self:custom.InstanceClassSmall}"

  routingToReplica:
    handler: route53.changeRecord
    environment:
      Msg: "${self:custom.MsgRoutingToReplica}"
      HostedZoneId: "${self:custom.HostedZoneId}"
      RecordName: "${self:custom.RecordName}"
      RecordValue: "${self:custom.ReplicaEndpoint}"


The Lambda functions are called with two different configurations for each, the first DNS to point to the master and then to re-point towards the replica

'use strict';
const AWS = require('aws-sdk');
const route53 = new AWS.Route53();
const sns = require('./lib/sns.js');

module.exports.changeRecord = (event, context, callback) => {

  route53.changeResourceRecordSets({
    ChangeBatch: {
      Changes: [{
        Action: 'UPSERT',
        ResourceRecordSet: {
          Name: process.env.RecordName,
          Type: 'CNAME',
          ResourceRecords: [{
            Value: process.env.RecordValue
          }],
          TTL: 60
        }
      }],
      Comment: "managed by lambda"
    },
    HostedZoneId: process.env.HostedZoneId
  }, function(err, data){
    if(err){
      sns.error("Cannot change DNS record "+process.env.RecordName+" to value "+process.env.RecordValue, function(){
        callback(err);
      })
    }else{
      sns.notify(process.env.Msg, callback)
    }
  });
};

for RDS for a scale-up and a scale-down to the starting size.

'use strict';
const AWS = require('aws-sdk');
const rds = new AWS.RDS();
const sns = require('./lib/sns.js');

module.exports.scale = (event, context, callback) => {

  rds.modifyDBInstance({
    DBInstanceIdentifier: process.env.InstanceIdentifier,
    DBInstanceClass: process.env.InstanceClass,
    ApplyImmediately: true
  }, function(err, data){
    if(err){
      sns.error("Cannot scale RDS instance "+process.env.InstanceIdentifier+" to "+process.env.InstanceClass, function(){
        callback(err);
      })
    }else{
      sns.notify(process.env.Msg, callback)
    }
  });

};

We've added message submissions to an SNS topic to be notified when the transactions took place and in case of problems be ready to intervene.

'use strict';
const AWS = require('aws-sdk');
const sns = new AWS.SNS();

module.exports.notify = (msg, callback) => {

  sns.publish({
    Message: msg,
    TopicArn: process.env.NotificationSNS
  }, callback);

};

module.exports.error = (error, callback) => {

  sns.publish({
    Message: process.env.MsgError +": "+ error,
    TopicArn: process.env.NotificationSNS
  }, callback);

};

We have a generic Lambda function attached on the topic capable of processing many types of incoming messages on a specific topic and forwarding them to a specific Slack Room.

Once the code was written, minimal and simple, and configured the Serverless project was enough to execute a single deploy command to have all the environment up&running.

Nothing remained to scheduled the Lambda executions. After numerous runs we made an estimate of how much time each step took and decided to roll out the executions using scheduled CloudWatch events with this type of configuration:

This give to the DNS 5 minutes to propagate (using a 60s TTL) and 10/12 minutes to the RDS instance to complete the scaling, scaling time plus the propagation time of the second DNS change.

We have analyzed the list of events launched by RDS and Route53 services so that we can connect to one or another and for us remain some doubts.

Changing the value of a DNS record involves waiting for its propagation and closing any existing database connections, there are no events for this, and it would not be possible to wait with a long polling until the modification is completed.

Modifying an RDS instance triggers some events during this type of modification, the first “DB instance shutdown” when the instance begins scaling, and then “DB instance restarted” at the instance restart. However, being a replica needs some extra time to finish synchronizing with the master and we are not quite convinced of an event-driven solution.

We will better analyze this solution if we go to this type of configuration.
Why not use this or that
We did not use a CPU usage or RDS memory alarm to trigger scaling due to the sudden increase of connections there was no time to scale and endure traffic.

Why not stop a second big replica and redirect the traffic on it after turning it on? As specified in AWS documentation, you can not stop reading a replica. If we want a larger or smaller instance, we just have to change a Lambda environment variable instead of scaling an instance in stop state.

We chose to scale instead of destroying and recreating because we use Terraform to manage the infrastructure we would have to import the new RDS instance id into the tfstate and this caused a few headaches.

Automatic Temporized Rds Scaling

Scaling Automatico Temporizzato Rds

Qualche settimana fa ho partecipato all’<a href="https://aws.amazon.com/it/awsomedaysitalia-2016/" target="_blank">AWSome Day</a> a Padova, evento gratuito organizzato da Amazon Web Service; si è trattato di una giornata di formazione e consulenza dagli esperti AWS, rivolta a IT Manager e sviluppatori che hanno iniziato a utilizzare le soluzioni di  Cloud Computing di Amazon o che, come me, sono (anzi – erano) totalmente ignoranti sull’argomento.

Il taglio dell’evento era sicuramente entry-level, quindi perfetto per me che conoscevo questa tecnologia solo per sentito dire e che sono curiosa di approfondire, anche perché AWS è la nostra infrastruttura di riferimento per i servizi di hosting evoluti.

Ho preso un sacco di appunti con l’intenzione di condividere i contenuti di questa giornata con alcuni colleghi che, come me, sentono parlare di AWS ma non sono mai entrati nel merito della soluzione. Alla fine ho pensato che il nostro primo strumento di diffusione della conoscenza, internamente ed esternamente all’azienda, è il blog quindi perchè non trasformare dei sommari appunti in un vero e proprio articolo?

Il Cloud Computing è una tecnologia che mette a disposizione, tramite la rete Internet, servizi basati sull’erogazione di risorse hardware e software con formula on demand.

Le risorse erogate comprendono, ad esempio: potenza di calcolo, spazio di archiviazione, strumenti di trasmissione dati e veri e propri servizi applicativi.

Un’azienda che necessita questo tipo di risorse per erogare i propri servizi online può appoggiarsi ad un provider di servizi cloud per acquistare e configurare un set di risorse, con la garanzia di poter ampliare questo set nel tempo in base alle esigenze di business e di capacità del sistema.

Le caratteristiche che hanno reso il Cloud Computing una tecnologia di successo sono:

<li style="font-weight: 400">
contenimento dei costi
</li>
<li style="font-weight: 400">
scalabilità del sistema
</li>
<li style="font-weight: 400">
garanzia della sicurezza
</li>

 principalmente si elimina l’esigenza di gestire un’infrastruttura IT personale, dall’acquisto dell’hardware e del software alla manutenzione del sistema passando dall’esigenza di avere in azienda figure specializzate. Inoltre il concetto di Cloud Computing si accompagna al concetto di flessibilità del costo in base alle reali risorse utilizzate (costo a consumo).

 possibilità di ampliare o diminuire le risorse a disposizione in base alla capacità richiesta per sopportare il carico garantendo stabilità al sistema; ad esempio incrementare le risorse a disposizione all’aumento della mole di dati da gestire, oppure all’aumento del traffico verso un servizio online.

 un buon service provider si occupa di mettere in atto tutte le pratiche necessarie per garantire il rispetto delle policy di sicurezza, la protezione dei dati memorizzati nel cloud e la conformità con una serie di certificazioni necessarie per la messa in opera di servizi che richiedono adeguati livelli di sicurezza (crittografia dei dati, autenticazione, antivirus, cifratura dei dischi ecc…)

Le principali tipologie di tecnologia Cloud sono così classificate:

<li style="font-weight: 400">
DaaS (Data as a service): il provider eroga e gestisce unicamente lo storage dati
</li>
<li style="font-weight: 400">
SaaS (Software as a service): il provider eroga e gestisce un servizio software
</li>
<li style="font-weight: 400">
IaaS (Infrastructure as a service): il provider eroga e gestisce l’infrastruttura di sole risorse hardware
</li>
<li style="font-weight: 400">
PaaS (Platform as a service): il provider eroga e gestisce l’infrastruttura hardware corredata dalle risorse software che permettono di avere una soluzione integrata per l’erogazione di applicazioni
</li>

Il cloud AWS è un sistema PaaS distribuito e disponibile in 190 paesi, suddivisi in aree geografiche (regions). Attualmente sono attive 14 regioni ma nel prossimo anno è prevista la nascita di altre 4.

Ogni regione comprende diverse zone di disponibilità (AZ – Availability Zone): le zone di disponibilità sono a tutti gli effetti dei data center distribuiti che permettono di replicare le applicazioni in modo da garantire scalabilità e disponibilità in caso di guasti su una particolare zona.

La scelta della regione da cui erogare i propri servizi e applicazioni dovrebbe essere dipendente dalla provenienza della maggior parte dei visitatori.

Componenti AWS, traduciamo un po’ di sigle

Nel sistema AWS quasi ogni componente o servizio è identificato, o meglio riconosciuto, da una sigla. Vediamo di far luce sulle principali.

E’ il primo componente nato in Amazon: si tratta di un servizio di archiviazione di risorse accessibili e condivisibili. E’ particolarmente adatto all’archiviazione di risorse statiche (storage, backup, media, componenti software da scaricare al boot per il delivery di macchine EC2).

<li style="font-weight: 400">
le risorse vengono archiviate all’interno di buckets, ovvero contenitori logici su cui è possibile impostare i permessi di accesso
</li>
<li style="font-weight: 400">
ad ogni risorsa salvata è associata una chiave all’interno del bucket
</li>
<li style="font-weight: 400">
le risorse archiviate sono disponibili via http e puntando alla url del servizio e alla chiave dell’oggetto e sono anche accessibili via rest/soap API tramite protocollo https
</li>
<li style="font-weight: 400">
non ha limiti di storage
</li>
<li style="font-weight: 400">
servizio scalabile, durevole e affidabile
</li>
<li style="font-weight: 400">
gestisce versioning e lifetime delle risorse
</li>
<li style="font-weight: 400">
le chiamate a S3 sono tutte crittografate con una chiave tramite un servizio
</li>
<li style="font-weight: 400">
si può attivare il logging di accesso alle risorse
</li>
<li style="font-weight: 400">
si possono configurare eventi (al get/put delle risorse)
</li>
<li style="font-weight: 400">
si paga lo spazio occupato
</li>

Per approfondimenti: <a href="https://aws.amazon.com/it/s3" target="_blank">https://aws.amazon.com/it/s3</a>

Si tratta dell’infrastruttura che consente di disporre on demand di istanze di macchine virtuali configurabili in base alle proprie necessità.

E’ possibile ridimensionare, accendere e spegnere le istanze in base all’esigenza di potenza di calcolo necessaria per far fronte al carico: la prima caratteristica è quindi la scalabilità in termini di capacità di carico.

Per attivare una macchina si può partire da un’immagine (

) o da un template precedentemente salvato, che è a tutti gli effetti un “modello di macchina”.

Si scelgono quindi il sistema operativo (Linux / Windows), l’availability zone in cui accendere la macchina, la dimensione della macchina selezionando famiglia (famiglie: T2, M3, M4, C3, C4, R3, G2, I2, D2), l’architettura della macchina configurando RAM, CPU, dischi di storage e rete.

Successivamente si procede alla configurazione dell’istanza specificando le regole di shutdown, il monitoraggio, i permessi di accesso e altre specifiche.

Il self-configuration della macchina mette a disposizione due importanti strumenti:

<li style="font-weight: 400">
Metadata: tutte le informazioni associate alla macchina. in qualsiasi momento si può verificare lo status dell’istanza interrogando i Metadata.
</li>
<li style="font-weight: 400">
User Data: informazioni da mandare all’istanza in modo che vengano eseguiti degli script al boot della macchina.
</li>

<li style="font-weight: 400">
alta scalabilità
</li>
<li style="font-weight: 400">
possibilità di sfruttare il servizio di Auto-Scaling: vengono accese in automatico delle istanze in base a logiche da noi definite; possiamo impostare metriche sulle risorse (ad es cpu) o su fasce orarie in base al traffico previsto
</li>
<li style="font-weight: 400">
accesso di root alla macchina
</li>
<li style="font-weight: 400">
possibilità di definizione di una chiave di accesso alla macchina via ssh
</li>
<li style="font-weight: 400">
massimo controllo della risorsa (sotto il profilo hardware e software)
</li>
<li style="font-weight: 400">
possibilità di gestire dei tag sulle risorse in modo da identificarle all’interno del cloud
</li>
<li style="font-weight: 400">
possibilità di configurare security groups
</li>
<li style="font-weight: 400">
possibilità di sfruttare il bilanciamento di traffico mediante Elastic Load Balancing
</li>
<li style="font-weight: 400">
usa hardware intel xeon sempre aggiornato
</li>
<li style="font-weight: 400">
si pagano solo le risorse utilizzate (dal momento in cui accendo la macchina)
</li>

Per approfondimenti: <a href="https://aws.amazon.com/it/ec2/" target="_blank">https://aws.amazon.com/it/ec2/</a>

Sono volumi di storage (dischi) montati sulle istanze EC2.

Risiedono nella medesima availability zone dell’istanza EC2 su cui vengono montati e sono particolarmente adatti come storage per dati che richiedono alta persistenza (indipendentemente dallo stato della macchina a cui sono collegati). Sono quindi adatti per l’utilizzo come file system primario o per lo storage di database o per memorizzare e gestire dati ad alta frequenza di lettura/scrittura.

Su una singola istanza EC2 possono essere montati diversi volumi EBS, un singolo volume EBS può però servire una singola istanza EC2 alla volta.

Per alti carichi di input/output su un disco si può scegliere la tipologia Provisioned IOPS SSD per garantire maggiori performance di I/O sul disco.

Tramite il servizio Amazon EBS Snapshot si possono gestire i backup dei dischi e bisogna fare attenzione alla tipologia di IP collegati, per garantire la persistenza degli indirizzi IP di riferimento in caso di azioni stop/start sull’istanza EC2 su cui è montato il volume.

Per approfondimenti: <a href="https://aws.amazon.com/it/ebs/" target="_blank">https://aws.amazon.com/it/ebs/</a>

Si tratta di un ambiente privato virtuale, logicamente isolato all’interno di AWS, in cui far risiedere le istanze EC2.

Di default Amazon crea una rete privata VPC per l’account utente, assegnando i servizi AWS attivi; è sempre possibile configurare le caratteristiche della rete VPC o crearne di nuove.

Altamente personalizzabile: è possibile selezionare gli intervalli degli indirizzi IP, configurare una VPN hardware con la propria rete aziendale e definire i permessi di accesso alla VPC.

E’ anche possibile creare delle subnet, o sottoreti: una subnet è sempre legata ad una availability zone.

La VPC prevede diversi livelli di sicurezza con possibilità di configurare:

<li style="font-weight: 400">
Security-groups: legati alle istanze EC2 lanciate all’interno della VPC. Funzionano come se fossero dei firewall che regolamentano il traffico outbound/inbound all’interno della VPC. Agiscono a livello di singola istanza EC2 e sono stateful.
</li>
<li style="font-weight: 400">
ACL per filtrare il traffico a livello di subnet, controllano il traffico in base al tipo di accesso (inbound/outbound) e sono stateless.
</li>

Per approfondimenti: <a href="https://aws.amazon.com/it/vpc/" target="_blank">https://aws.amazon.com/it/vpc/</a>

Bitbull

Aws

Ingress rules for dynamic IPs

Sul perché quest’anno saremo presenti al Netcomm e perché lo saremo insieme ad Amazon Web Services

My serverless journey

Automatic Temporized Rds Scaling

AWS for dummies