Management of Security Group ingress rules to allow access to dynamic IPs.

Our infrastructures are designed to remain isolated in a private subnet of the VPC, this in order to increase the security level and block any type of unwanted access directly from network layer. For various management reasons like verification and troubleshooting we need to access the resources inside these isolated infrastructures.

The common solution for this type of setup is the 

, a world-exposed EC2 instance that allows access to other isolated resources. To block any unwanted access we use a whitelist system through the ingress rules of the security group of the bastion instance. Any SSH access is blocked except for certain IPs which are manually added when requested.

This system is very robust: it delegates access control to the security group, a resource managed by AWS, by doing so we do not have to take on the charge of system configurations (for example iptables) and manage various and possible security patches.

This solution is great when using static IPs, a request is made to the DevOps team to whitelist a certain IP and this is added among the ingress rules for the specific port of service requested (port 22 for SSH access). This works well when the access request comes from other infrastructures or offices or coworking that have a static IP.

Since our company is distributed and adopting the 

, developers or DevOps team members often connect from networks with dynamic IPs. This means we have to continuously manage security group rules of the bastion instance by removing and adding IPs almost every day.

After evaluating some solutions, we decided to adopt a serverless system based on API Gateway and Lambda that automatically adds the IP to whitelist on the security group. To dig more into detail: we have exposed an HTTP API using AWS API Gateway service that, upon an HTTP request id received, it trigger the execution of a Lambda. The Lambda function retrieves the IP of the caller from the input event and modifies security group ingress rules to allow access for SSH service.

This system is very simple and being based on serverless services, with very low traffic, the cost is zero. The Lambda code is also very simple:

As you can see, to verify the origin of the request we opted a solution that pass a certain token as a path parameter. This is because we want the use of this endpoint to be as simple as possible and in this way we can directly open a link similar to this:

https://xxxxxxxxx.execute-api.us-east-1.amazonaws.com/stage/whitelist/yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

directly in the browser without having to set a particular header (for example 

 using the API Gateway build-in system) that would need, for example, a curl call from the terminal.

The configuration of this token, security group ID, protocol, port and description is done through environment variables, in this way we can deploy the same Lambda code for different environments and infrastructures.

 (incrementable up to 1000 via a request to AWS) which may be affected after a few months of system usage. In addition, this automatic system is used exclusively by those who connect using a dynamic IP, this means that the IP will be reassigned to someone else and must therefore be removed from whitelist.

The solution we have adopted is to add another Lambda, this time triggered by a scheduled CloudWatch event, which will clean up the security group rules.

Obviously we do not want the rules added manually for static IPs to be removed as well, otherwise it would cause an inadvertent blocking of third-party integration accesses. To solve this problem we use as discriminant the rule's description, set through the environment variable 

 in the Lambda described above. In this way we are going to clean up only the rules added through this automatic system and allowing the DevOps team to manage some rules manually.

The Lambda code is also very simple in this case:

 API by passing the list of ingress rules with a specific description.

This system meets our needs and has a good compromise between security and usability. This system only grants access by a certain IP to the SSH service exposed by the bastion but does not grant access to the machine, whoever wants to connect must still use their own personal SSH key.

Ingress rules for dynamic IPs

Regole di ingresso per IP dinamici

? I've heard about it so many, too many maybe,so I decided to embark on this journey to discover this new technology.

So I prepare the backpack, the boots, rope, harness and I start..

Outside is a gloomy, autumn day, the right climate to stay in houses with a cup of coffee and my notebook. Click on my faithful companion Firefox, where to start? First of all I have to choose which cloud provider to use ..5 seconds to think.. AWS of course.

The first research leads me to hundreds and hundreds of blog articles, really so much material. I try to make a summary so I convince myself to have understood:

Do you know servers? Those strange computers, in dark rooms, ok, out of fashion anymore... why? They are difficult to manage, you have to use a thousand precautions, uninterruptible power supply in case the light goes out, the right temperature, absolutely no dust, fire doors, fire systems with argon etc..etc.. so to ensure a stable service without become crazy and spend a lot of money services like AWS, Google Cloud, Azure and many others "rent" part of their data center, you only have to configure the infrastructure with a few clicks and only worry about the software part.

After this great step forward there are other problems (and of course nobody likes them), you have to keep up-to-date the operating system installed on your virtual servers, install security patches, configure a scaling of the computational resources, redundate the infrastructure in order to guarantee an high reliability of the service.. and all that, so what have the cloud providers invented to come up against? Manage some of the services like MySQL, Elasticsearch, Redis, RabbitMQ and many others, directly accessing it without having to take care of installing and updating these services, obviously the configuration (even if a little limited) is always possible.

After this last step still remains the problem of virtual machines that will run your software (always the same problem mentioned above) so what is the next step? The Serverless! That is, managed platforms where the only thing you need to worry about is your code, nothing else, it's the cloud provider that will automatically take care of scaling, updating the operating systems that host your code and ensuring its reliability.

Server-less does not mean that there are no servers but that you do not need to manage them :)

After the initial study done yesterday, today is the day of the practical test! There are many guides that explain how to create a simple API using AWS services, but the activation steps of the services are done directly from the web console (or using CLI), however convenient it does not allow you to easily reproduce the configuration steps and the changes are not traced. AWS provides a service called 

, by describing the infrastructure with a JSON or YAML file you can provision all the resources you need in one go, manage updates and rollback if an error occurs.

Of course, writing a huge JSON or YAML file is not very convenient, so I looked for what AWS framework provides:

Honestly I found a bit of confusion about what to use but the advice is: SAM for API services, Mobile CLI to avoid, Amplify for more complex applications also equipped with frontend. I then focused on the latter, to see the full experience and see the news.

Very very easy to use, maybe even too much, just download the cli

enter the root folder of your web application and add Amplify as an addiction

and as a last part initialize the CloudFormation base stack where just a couple of access rules will be created and little else

Now the fun part comes up, add services that our application needs, the Amplify CLI will always use guided steps to configure them. By making an overview of what we can add, we see:

Wow! There is practically everything and you can add one command

and once the selected service has been set up, deploy the changes

Really nice, some services I already knew them as 

Amplify is really a great tool, especially for the integrations for React, AngularJS and Vue.js that allow you to connect services quickly using the framework that we like best. It took 15 minutes to get a simple application (yes, the usual old TODO list) up&running, but if you do not get your hands dirty by putting them in the engine, I do not feel completely satisfied.

I then start to integrate my application written in 

 starting from authentication. The configuration of Cognito is really simplified:

1 minute and everything is ready to manage the users.

I am amazed at how much functionality Cognito offers, confirmation of the email and/or telephone number sending you a code (via SMS for the phone), login with Google, Facebook, Amazon and OpenID, and provide a complete self-managed OAuth2 server. To create this, from scratch, with a PHP server or nodeJS at least 1/2 days of work would be served, for this I spent 1 minute, it's crazy.

Now the database, I had already used in the past DynamoDB for some additions but never in combination with 

, a fairly recent service (no more than 1 year and a half) that helps you manage access to your data using GraphQL endpoints. It is appropriate to say: the latest fashion!

With Amplify it was really simple, you certainly need to have some experience with GraphQL.

? Please select from one of the below mentioned services (Use arrow keys)
❯ GraphQL
 REST

not only GraphQL endpoints but also REST! a step forward to choose the authentication method

? Please select from one of the below mentioned services GraphQL
? Provide API name: myNotesApi
? Choose an authorization type for the API (Use arrow keys)
❯ API key
 Amazon Cognito User Pool

you can use either static keys (generated during this process) or using the authentication system created first with Cognito, obviously I choose the last option.

Today I started writing models for my application, one of Amplify's most convenient features is the automatic generation of GraphQL resolvers, even with 

 to specify the access level. This last part is vital to restrict user access only to their data and not to the entire DynamoDB table.

Here the first problem, I tried to faithfully follow the documentation but nothing, checking the resolvers generated I saw no difference using or not the transformer 

. I lost a lot of time on this until, I gave up, I created an 

 on the Github respository to ask for support, they really answered me rapidly and it turned out that actually it was not really working... okay, things that can happen.

During debugging I discovered that the DynamoDB table indexes were created with the 

 for Reads and Writes to 5, without possibility of being modified.

For those who do not know, this value is one of the parameters that is used to calculate the cost of the service of DynamoDB: the number of accesses in reading or writing per second (not so simple but I do not enter into the specification). In order not to go into too much detail, this non-relational database answers for each query at the maximum possible speed regardless of the amount of data and there is no limit to the number of lines (pay for the weight in GB of the table). The negative part is a somewhat tricky query engine and you need to carefully create table indexes that are vital for sorting and querying.

I did not expect that for any test you will pay at least around $13, a bit expensive considering that with less than $1 I have 2 projects of IoT, a GoogleHome action, my website (with two environments) and I use S3 to make an encrypted backup of some important files. With just 5/6 models I arrived at disproportionate figures, with a DynamoDB table ready for production when the application was not born yet. I then opened another 

, this time not yet taken into consideration for now.

So I eliminated the project and I got lost, I had found a very easy to use and powerful tool but it's still too young to be used for production. Being so simple it does not even allow such a deep changes of some parts of it except with different workarounds.

Several nights are spent evaluating alternatives.

Over the days I have collected several alternatives to Amplify, difficult because there are no other CLI that with a couple of commands do EVERYTHING.

, a very powerful and modular framework with a large community behind which develops hundreds of plugins. I've already used it before for some cloud infrastructure and API automation projects and I've always been comfortable with it.

The management of the resources directly uses the CloudFormation templates, this guarantees to be always updated and uses the functionalities served by AWS without having to report the changes within the framework (an example, 

 uses its syntax and its integration, can not keep up and I always found dozens of bugs).

When there are many resources to add out of those managed by Serverless the 

 explains how to create them in the file, not make the mistake of putting everything in a single file of 2000 lines that is difficult to maintain, better divide it into multiple files

resources:
 - ${file(resources/cognito.yml)}
 - ${file(resources/userRole.yml)}
 - ${file(resources/queues.yml)}
 - ${file(resources/tables/todos.yml)}
 - ${file(resources/tables/projects.yml)}
 - ${file(resources/tables/categories.yml)}
 - ${file(resources/outputs.yml)}

and then in these files specify the resources

Resources:
 AccountTable:
   Type: AWS::DynamoDB::Table
   Properties:
     TableName: "${self:service}-${self:provider.stage}-${self:custom.todosTableName}"
     AttributeDefinitions:
       -
         AttributeName: userId
         AttributeType: S
       -
         AttributeName: id
         AttributeType: S
     KeySchema:
       -
         AttributeName: userId
         KeyType: HASH
       -
         AttributeName: id
         KeyType: RANGE
     ProvisionedThroughput:
       ReadCapacityUnits: 1
       WriteCapacityUnits: 1
     StreamSpecification:
       StreamViewType: "NEW_AND_OLD_IMAGES"

in this way the code will be much more organized and the files smaller and more maintainable.

The previous speech also applies to the configurations of the project, better specify a separate file so you can change it depending on the environment or project without using hundreds of environment variables

service: MyService
custom: ${file(./config.yml)}

provider:
 name: aws
 runtime: nodejs8.10
 stage: ${opt:stage,self:custom.env}
 region: ${self:custom.region}

the configuration file becomes much easier to view

env: "Stage"
region: "eu-west-1"

todosTableName: "Todos"

Attracted by Amplify to use GraphQL I immediately started using 

, easy to use and, with some organization and re-use of resolvers, I arrived at a good point. I realized that I was writing so much code that meant keeping it, testing it, optimizing it, and taking a lot of time to create a simple CRUD feature for the models.

A doubt has arisen: do I really need a GraphQL APIs? Of course they are really comfortable, expressive and easy to keep up to date but why? More specifically: do I need APIs to consume? The APIs are mainly used to access the database, add server-side logic (which the user can not manipulate) and make the service available to third-party services.

DynamoDB has its own build-in functionality to limit access to rows, columns and operations based on the user, ie a user can only see certain attributes of the saved object (the "columns"), access only the objects that he has created (the "lines") and read some tables without the possibility of modifying it this by identifying the user through Cognito authentication.
What does it mean? Which has a security system for CRUD access already served, without having to make these checks via code, it is already done! It's AWS to guarantee this security, do you want to be a developer to do better than the team of 100 Amazon developers? You could, but in 99% of cases you're missing some vulnerabilities.

You can say everything about the AWS ecosystem according to the opinions, that is inconvenient, expensive, difficult, complicated but it can not be said that it's not safe, so ok, I give them this part.

Side note: once authenticated a user with Cognito can access any AWS resource limiting access and operations through the 

, not all resources support a granularity like DynamoDB but already only for file upload the S3 service can do it (think about the user's avatar, for example, convenient!).

I need it? What am I doing that I have to hide from the user? If I already know that he will only access his lines at most is breaking the formatting of his data, well, he will not be able to view them correctly.

Surely soon or later there will be the need to call third party services to add some functionality, no problem, with 

 where you can do whatever you want, the events that can trigger this custom logic are endless: from creating or editing an object in DynamoDB to uploading a file to S3, even all the flow of authentication/registration/confirmation of Cognito users, no more.

Do you need it already? Do we already have partners who ask to be able to take advantage of our services? if the answer is yes, you can create the entire API compartment you want with Serverless 

 creating only the endpoints you need, otherwise you can skip this part.

 is an AWS service that allows you to organize API endpoints and manipulate request and response parameters.

No, I do not need them for the moment. I wanted to give Amplify another chance and use the client directly to access DynamoDB and use the AWS resources I need. Very clean and really simple to use.

In any case, for authentication to the API endpoint you can use either Cognito directly (useful in case calls from the same application that you need to login) of the fixed keys and an access key generated 

from the OAuth2 endpoint served by Cognito

. This last option makes the endpoint very professional by using the security of an OAuth2 access and allows configuring custom scopes to regulate access to resources.

Once the backend part was put in place, it was time to start creating the frontend part. I did not have much doubt about the technology to use, having enough experience with 

and following the wizard I chose a very simple configuration and then saved as a preset. The configuration to insert in the 

{
 "packageManager": "npm",
 "useTaobaoRegistry": false,
 "presets": {
   "vue-router-vuex": {
     "useConfigFiles": true,
     "plugins": {
       "@vue/cli-plugin-babel": {},
       "@vue/cli-plugin-pwa": {},
       "@vue/cli-plugin-eslint": {
         "config": "base",
         "lintOn": [
           "commit"
         ]
       }
     },
     "router": true,
     "routerHistoryMode": true,
     "vuex": true,
     "cssPreprocessor": "stylus"
   },
 }
}

? It was a choice driven by the component library I want to use: 

. I've already used it for some projects and I've always felt good, I find it very complete for the quantity of components and, very important thing: very customizable.

Here too, thanks to the CLI, a command was enough to add this library along with some sample configurations

My serverless journey

Il mio viaggio nel mondo serverless

Temporized RDS read replica AutoScaling with Lambda and Route53

One of our hosting infrastructure on AWS recently has encountered some problems due to a huge and rapid increase in traffic during some pre-defined periods, at the same time as a newsletter, that quickly brought many more users to visit our customer’s Magento e-commerce.

The number of requests made to our ElasticLoadBalancer ranged from 400 to 3,000 per minute in less than 5 minutes, that did not give us enough time to scale-up the infrastructure before it fall down.
From our analysis we have identified the bottleneck in the database, especially in categories and catalog read queries, dynamics based on various configurations and based on the connected user.

To better control the load on the database, we have decided to use two RDS instances, a master for write operations, and a replica for read operations to only scale the necessary part and control costs.
We noticed an improvement but the same reading replication suffered from the load and they quickly saturated the database connections.

To prepare us for incoming traffic, one hour before the peak of traffic, we changed the database endpoint from the Magento configurations to not use the replication. When the scaling was done we brought the configuration back to the previous one.

This solution worked fine but it was very uncomfortable to change the Magento configurations, so we had to empty the cache and this would be a further burden on the first few site visits in addition to having to manually process the procedures with the risk of human error.

The solution we adopted for not having to modify the connection configurations of the Magento database endpoint was to use internal VPC DNS managed with Route53 by mapping the hosts of RDS instances with domain names using CNAME records in the magento configurations file.

This way we have the control of pointing the web application to the database agnostically to its configurations. This allows us to redirect the read replication connections to the master during scaling and this not generate any downtime.

Once we have reached a stable solution that allows us to scale without downtime, it only remains to set this procedure automatically.

"This is a task for Lambda," we said, so we wrote two Lambda functions in NodeJS that are able to modify the RDS instance and the Route53 DNS record attached to its endpoint.
For this small project we chose to use the Serverless framework that we used in the past with great satisfaction.

functions:
  routingToMaster:
    handler: route53.changeRecord
    environment:
      Msg: "${self:custom.MsgRoutingToMaster}"
      HostedZoneId: "${self:custom.HostedZoneId}"
      RecordName: "${self:custom.RecordName}"
      RecordValue: "${self:custom.MasterEndpoint}"

  scaleUp:
    handler: rds.scale
    environment:
      Msg: "${self:custom.MsgScaleUp}"
      InstanceIdentifier: "${self:custom.InstanceIdentifier}"
      InstanceClass: "${self:custom.InstanceClassBig}"

  scaleDown:
    handler: rds.scale
    environment:
      Msg: "${self:custom.MsgScaleDown}"
      InstanceIdentifier: "${self:custom.InstanceIdentifier}"
      InstanceClass: "${self:custom.InstanceClassSmall}"

  routingToReplica:
    handler: route53.changeRecord
    environment:
      Msg: "${self:custom.MsgRoutingToReplica}"
      HostedZoneId: "${self:custom.HostedZoneId}"
      RecordName: "${self:custom.RecordName}"
      RecordValue: "${self:custom.ReplicaEndpoint}"


The Lambda functions are called with two different configurations for each, the first DNS to point to the master and then to re-point towards the replica

'use strict';
const AWS = require('aws-sdk');
const route53 = new AWS.Route53();
const sns = require('./lib/sns.js');

module.exports.changeRecord = (event, context, callback) => {

  route53.changeResourceRecordSets({
    ChangeBatch: {
      Changes: [{
        Action: 'UPSERT',
        ResourceRecordSet: {
          Name: process.env.RecordName,
          Type: 'CNAME',
          ResourceRecords: [{
            Value: process.env.RecordValue
          }],
          TTL: 60
        }
      }],
      Comment: "managed by lambda"
    },
    HostedZoneId: process.env.HostedZoneId
  }, function(err, data){
    if(err){
      sns.error("Cannot change DNS record "+process.env.RecordName+" to value "+process.env.RecordValue, function(){
        callback(err);
      })
    }else{
      sns.notify(process.env.Msg, callback)
    }
  });
};

for RDS for a scale-up and a scale-down to the starting size.

'use strict';
const AWS = require('aws-sdk');
const rds = new AWS.RDS();
const sns = require('./lib/sns.js');

module.exports.scale = (event, context, callback) => {

  rds.modifyDBInstance({
    DBInstanceIdentifier: process.env.InstanceIdentifier,
    DBInstanceClass: process.env.InstanceClass,
    ApplyImmediately: true
  }, function(err, data){
    if(err){
      sns.error("Cannot scale RDS instance "+process.env.InstanceIdentifier+" to "+process.env.InstanceClass, function(){
        callback(err);
      })
    }else{
      sns.notify(process.env.Msg, callback)
    }
  });

};

We've added message submissions to an SNS topic to be notified when the transactions took place and in case of problems be ready to intervene.

'use strict';
const AWS = require('aws-sdk');
const sns = new AWS.SNS();

module.exports.notify = (msg, callback) => {

  sns.publish({
    Message: msg,
    TopicArn: process.env.NotificationSNS
  }, callback);

};

module.exports.error = (error, callback) => {

  sns.publish({
    Message: process.env.MsgError +": "+ error,
    TopicArn: process.env.NotificationSNS
  }, callback);

};

We have a generic Lambda function attached on the topic capable of processing many types of incoming messages on a specific topic and forwarding them to a specific Slack Room.

Once the code was written, minimal and simple, and configured the Serverless project was enough to execute a single deploy command to have all the environment up&running.

Nothing remained to scheduled the Lambda executions. After numerous runs we made an estimate of how much time each step took and decided to roll out the executions using scheduled CloudWatch events with this type of configuration:

This give to the DNS 5 minutes to propagate (using a 60s TTL) and 10/12 minutes to the RDS instance to complete the scaling, scaling time plus the propagation time of the second DNS change.

We have analyzed the list of events launched by RDS and Route53 services so that we can connect to one or another and for us remain some doubts.

Changing the value of a DNS record involves waiting for its propagation and closing any existing database connections, there are no events for this, and it would not be possible to wait with a long polling until the modification is completed.

Modifying an RDS instance triggers some events during this type of modification, the first “DB instance shutdown” when the instance begins scaling, and then “DB instance restarted” at the instance restart. However, being a replica needs some extra time to finish synchronizing with the master and we are not quite convinced of an event-driven solution.

We will better analyze this solution if we go to this type of configuration.
Why not use this or that
We did not use a CPU usage or RDS memory alarm to trigger scaling due to the sudden increase of connections there was no time to scale and endure traffic.

Why not stop a second big replica and redirect the traffic on it after turning it on? As specified in AWS documentation, you can not stop reading a replica. If we want a larger or smaller instance, we just have to change a Lambda environment variable instead of scaling an instance in stop state.

We chose to scale instead of destroying and recreating because we use Terraform to manage the infrastructure we would have to import the new RDS instance id into the tfstate and this caused a few headaches.

Automatic Temporized Rds Scaling

Scaling Automatico Temporizzato Rds

Today I will tell you my experience about the configuration of a Bitbucket pipeline to compile static assets and DI (Dependency injection) and then, run the deployment on the webserver.

What are we talking about? Pipeline, Continuous Integration (CI) and Continuous Delivery (CD) are terms that you may have already heard in DevOps world, but also if you are developers soon or later you’ll hear it.

 term means the system where the source code of an application is downloaded from the repository and a series of steps or tasks are performed. This is done in an isolated environment (usually using a Docker container) that should be as close as possible to the one that will host the actual application.

 is the automatic procedure that brings the changes from a repository (version control server) towards a common environment, usually the staging environment. This operation is also performed several times a day and often test functions are connected to validate the work. This prevents misalignment and incompatible versions of various branches of development and its external libraries.

 instead is a procedure that compile code from a repository and and deploy the application to the end user or production web server through various steps.

Using Magento version 1.x many of you surely have configured a pipeline, with any CI/CD service as Bitbucket, Gitlab, Jenkins, Trevis or any other service with no major problems. The key part that facilitates this work is the use of composer for the installation of modules using 

magento-hackathon/magento-composer-installer

With Magento 2.x you certainly had to deal with the deployment of static asset, compilation of DI and many optimizations.

This requires steps after installation of the modules, of course we try to perform this procedure in the pipeline not to load the web servers during the deployment. In the current version (2.1.7) to execute the commands needed to compile require database connection, this adds a complication: being the pipeline in isolated container, it can not connect to the target environment and database, also It wouldn’t be a good practice.

The first requirement is a repository on Bitbucket, if you use other services like GitLab you can use their pipeline system, very well done, or in case you are using GitHub you need an external service.

Add the Bitbucket the pipeline configuration file, called “bitbucket-pipelines.yml”.

The file structure is very simple, on the first line we indicate which Docker image will be used to run the pipeline containers, in our case we baked an image very similar to production webserver and local system called 

Now we can add the statement of the pipeline, there are two possibilities: use it automatically, when any commit is submitted (filterable with tags or branch) or let us start it with a simple click. In most cases there is a single staging server and you don’t want accidentally overwrite someone else’s changes, which currently is testing a feature. We should follow the second plan.

We declare that we want a manual pipeline

custom:
    01_staging:

    02_production:

We now declared two pipelines, one for each environment. One useful tip is to prefix the name with an incremental number, this is because Bitbucket will show them in alphabetical order and we don’t want the first selected value is “Production” instead of “Staging”.

Then we add the commands that will be executed, doing it in “step” section (currently only one step at pipeline is supported, but in the future will be able to run them more than one in parallel) and we use environment variables to make all the things more configurable. A tip is to use the same name in scripts and enhance them based on the target environments that we want to use.

01_staging:
      - step:
          script:
            - export LOCALES="en_US it_IT"
            - chmod +x deploy-scripts/Build.sh && ./deploy-scripts/Build.sh

Bear in mind that all commands are executed from a file stored in the repository, this to make it more readable.

You can configure the environment variables from the settings panel and obfuscate those at risk such as the user’s AWS credentials.

So, let’s write commands required from compiling process creating the file 

#!/usr/bin/env bash

set -e # interrompe l’esecuzione in caso di errore

TAG=$BITBUCKET_COMMIT

#Configurazione di composer

COMPOSER_AUTH=$(cat &lt;&lt;EOF { "github-oauth": { "github.com": "$COMPOSER_AUTH_GITHUB_TOKEN" }, "http-basic": { "repo.magento.com": { "username": "$COMPOSER_AUTH_MAGENTO_REPO_USERNAME", "password": "$COMPOSER_AUTH_MAGENTO_REPO_PASSWORD" } } } EOF ) mkdir -p /root/.composer && echo "$COMPOSER_AUTH" &gt; /root/.composer/auth.json

composer install --no-dev

With these commands we will install all Magento modules and the core itself, I recommend to put the .lock composer file under version control to avoid any problem during this step by knowing in advance which versions will be installed.

At this point something is missing, the database connection! Bitbucket allows us to create a service, namely a second docker container, connected to the one that executes the steps.

Let’s go back to the configuration file and add to the bottom of bitbucket-pipelines.yml file as follows

definitions:
  services:
    mysql:
      image: mysql
      environment:
        MYSQL_DATABASE: 'magento'
        MYSQL_ROOT_PASSWORD: 'root'

In this way, we are adding a container with a MySQL instance with a database already created named “magento” with root user’s password populated.

We add this service to the declaration of pipeline

01_staging:
      - step:
          script:
            ...
          services:
            - mysql

Now we can connect to our temporary database and load a dump taken from the staging and cleaned up by log tables.

# Connessione al database ed esecuzione del dump

M2_DB_HOSTNAME=127.0.0.1
M2_DB_NAME=magento
M2_DB_USERNAME=root
M2_DB_PASSWORD=root

aws s3 cp s3://$CODEDEPLOY_S3_BUCKET/$CODEDEPLOY_ENVIRONMENT/dump.sql /tmp/dump-$TAG.sql # scarichiamo da un bucket S3 il file di dump
mysql -h $M2_DB_HOSTNAME -u $M2_DB_USERNAME -p"$M2_DB_PASSWORD" $M2_DB_NAME &lt; /tmp/dump-$TAG.sql # applichiamo il dump

ENV_FILE=$(cat &lt;&lt;EOF ADMIN_FRONTNAME="admin" APP_KEY="$APP_KEY" SESSION_SAVE="files" DB_HOSTNAME="$M2_DB_HOSTNAME" DB_NAME="$M2_DB_NAME" DB_USERNAME="$M2_DB_USERNAME" DB_PASSWORD="$M2_DB_PASSWORD" MAGE_MODE="production" CACHE_MODE="disable" WEBSERVER_USER="www-data" EOF ) echo "$ENV_FILE" &gt; .env
composer run-script generate-config # usiamo un modulo sviluppato da noi per creare il file di configurazione env.php 

Now Magento is connected to the database and we can execute the CLI  commands to run compiling process

# Magento deploy
php bin/magento module:enable --all # abilitiamo tutti i moduli

php bin/magento setup:upgrade
php bin/magento setup:di:compile
php bin/magento setup:static-content:deploy $LOCALES

rm app/etc/env.php
rm .env
rm -r var/cache/*
rm -r .git/
rm docker-compose.yml
rm Vagrantfile

An essential part of the CI / CD is the creation of an artifact, which is a file including the version of the application ready to run.

So, our version is ready for deployment. For this last step in Bitbull we use a special AWS service called 

 that, with some configuration, allows us to bring the new version on our servers. This is possible with few extra lines.

aws s3 cp /tmp/release-$TAG.zip s3://$CODEDEPLOY_S3_BUCKET/$CODEDEPLOY_ENVIRONMENT/release-$TAG.zip
aws deploy register-application-revision --application-name $CODEDEPLOY_APP_NAME --s3-location bundleType=zip,bucket=$CODEDEPLOY_S3_BUCKET,key=$CODEDEPLOY_ENVIRONMENT/release-$TAG.zip --region $AWS_REGION

aws deploy create-deployment --application-name $CODEDEPLOY_APP_NAME --deployment-group-name $CODEDEPLOY_DEPLOYMENT_GROUP --s3-location bundleType=zip,bucket=$CODEDEPLOY_S3_BUCKET,key=$CODEDEPLOY_ENVIRONMENT/release-$TAG.zip --region $AWS_REGION

This command is asynchronous, so the pipeline will end without waiting CodeDeploy, to monitor your progress, and check the presence of any errors you need to access the AWS console. We have connected CodeDeploy notifications to our Slack to be notified in case of errors.

Save the script file and pipeline configuration file, then you have to do a commit.

Now our pipeline is ready, we can run it.

We go on Bitbucket and choose which commit to starting from the pipeline, you can also select a branch and will last commit on it will be used..

You will find on your right the command “Run pipeline”, click on it and Bitbucket will show you all the pipelines declared in the configuration file.

Once it is launched a special control panel will open, it shows the output of the commands; In the case of errors, this is the place to check out what the problem is.

During execution on the left side will appear a “stop” button that can the execution, in the case you launched it by mistake. At the end, however, with the “rerun” button you can run it again starting from the same commit.

To speed up the downloading of composer modules it is convenient to use the pipeline cache. Be careful to use the default served from Bitbucket, otherwise all modules under /vendor directory will be cached and there will be no Magento core symlinks created by composer during the install command. In Bitbull, we use a solution that helps partially, but not yet optimized.

At the bottom of the bitbucket-pipelines.yml files add the following lines

definitions:
  services:
    ...
  caches:
    composercache: ~/.composer/cache

and then connect it to the pipelines to apply the configuration

01_staging:
      - step:
          caches:
            - composercache


If you use Slack as a messaging app you’ll be more comfortable creating a WebHook to send notifications in the project channel.

Pipeline Magento2

 il 12 e 13 Marzo siamo stati a Verona al 

, un evento tutto incentrato sul mondo PHP (come si può intuire dal nome dell’evento).

Il primo intervento che ci ha dato il benvenuto è stato tenuto da 

, molto molto interessante. Anche se in Bitbull non facciamo alcuna distinzione tra uomo o donna come se fosse una cosa normalissima, è sempre bello sentire che la comunità di sviluppatori ha così tanto a cuore la questione, molto bello l’accenno alle giovani leve e come “non insegnare la discriminazione”.

Anche se gran parte della platea conosceva già l’argomento siamo stati tutti molto presi e attenti fino alla fine dell’intervento, chiusosi con un grandissimo applauso.

Uno degli interventi che mi ha lasciato più dubbi è stato quello di 

 intitolato Feature Toggles. Facendo un riassunto ha spiegato come impostare un sistema dove è possibile disabilitare ed abilitare le feature introdotte nella propria applicazione con dei check di costanti globali. Molto comodo durante lo sviluppo e quando alcune funzionalità in produzione devono essere disabilitate e riabilitate senza dover rifare il deploy, però trovo che sporchi molto il codice inserendo centinaia di if else che se un domani si vogliono poi rimuovere bisogna farlo a mano uno per uno!

Abbiamo partecipato con molto entusiasmo al workshop “Basic CQRS and Event Sourcing with Prooph” guidato da 

. Qui devo però fare un appunto, in questo evento ho visto molti tentativi di trasformare il normale workflow sincrono di PHP in asincrono e l’ho trovata una grande forzatura.

A parere mio il PHP va bene così com’è, se si ha il bisogno di usare eventi e funzioni non bloccanti ci sono molti altri linguaggi più adatti a questo scopo.

Al termine dell’evento, l’after party ha dato la possibilità di fare networking e conoscere molte persone interessanti, scambiare pareri e confrontarsi sulle tecnologie utilizzate.

Il keynote della seconda giornata è stato senza dubbio l’intervento più discusso di tutti dal titolo “NoEstimates”. Vasco Duarte ci ha illustrato come gestire un progetto cercando di diminuire i problemi derivanti dalle stime non sempre precise e soggette ad imprevisti.

Bitbull

Fabio Gollinucci

Posts written by Fabio Gollinucci

Ingress rules for dynamic IPs

My serverless journey

Automatic Temporized Rds Scaling

Pipeline Magento2

PHPDAY 2017

Electron – Una nuova era per le applicazioni desktop