Ingress rules for dynamic IPs

Management of Security Group ingress rules to allow access to dynamic IPs.

Necessity

Our infrastructures are designed to remain isolated in a private subnet of the VPC, this in order to increase the security level and block any type of unwanted access directly from network layer. For various management reasons like verification and troubleshooting we need to access the resources inside these isolated infrastructures.

Bastion instance

The common solution for this type of setup is the use of a bastion, a world-exposed EC2 instance that allows access to other isolated resources. To block any unwanted access we use a whitelist system through the ingress rules of the security group of the bastion instance. Any SSH access is blocked except for certain IPs which are manually added when requested.

This system is very robust: it delegates access control to the security group, a resource managed by AWS, by doing so we do not have to take on the charge of system configurations (for example iptables) and manage various and possible security patches.

Static and dynamic IPs

This solution is great when using static IPs, a request is made to the DevOps team to whitelist a certain IP and this is added among the ingress rules for the specific port of service requested (port 22 for SSH access). This works well when the access request comes from other infrastructures or offices or coworking that have a static IP.

Since our company is distributed and adopting the smart working, developers or DevOps team members often connect from networks with dynamic IPs. This means we have to continuously manage security group rules of the bastion instance by removing and adding IPs almost every day.

Automatic rules management

After evaluating some solutions, we decided to adopt a serverless system based on API Gateway and Lambda that automatically adds the IP to whitelist on the security group. To dig more into detail: we have exposed an HTTP API using AWS API Gateway service that, upon an HTTP request id received, it trigger the execution of a Lambda. The Lambda function retrieves the IP of the caller from the input event and modifies security group ingress rules to allow access for SSH service.

This system is very simple and being based on serverless services, with very low traffic, the cost is zero. The Lambda code is also very simple:

const AWS = require('aws-sdk');
const ec2 = new AWS.EC2({
  apiVersion: '2016-11-15',
  logger: console
});

/**
 * Configurations
 */
const ROLE_DESCRIPTION = process.env.ROLE_DESCRIPTION || 'managed by lambda';
const SECURITY_GROUP_ID = process.env.SECURITY_GROUP_ID
const API_TOKEN = process.env.API_TOKEN
const PROTOCOL = process.env.PROTOCOL || 'tcp'
const PORT = process.env.PORT || 22

/**
 * Lambda handler
 */
module.exports.handler = async (event) => {
  const sourceIp = event.requestContext.identity.sourceIp;
  
  // Check authorization using token in path parameters
  if(event.pathParameters.token && event.pathParameters.token.toString().trim() === API_TOKEN){
    console.log('Checking authorization for source IP '+sourceIp+' on security group '+SECURITY_GROUP_ID+'..')
    try {

      // Authorize IP on security group
      await ec2.authorizeSecurityGroupIngress({
          GroupId: SECURITY_GROUP_ID,
          IpPermissions: [
            {
                IpProtocol: PROTOCOL,
                FromPort: PORT,
                ToPort: PORT,
                IpRanges: [
                  {
                    CidrIp: sourceIp + '/32',
                    Description: ROLE_DESCRIPTION
                  }
                ]

            }
          ]
      }).promise()
    } catch (err) {
      if (err.code == 'InvalidPermission.Duplicate') {
        // Return invalid request
        return {
          statusCode: 409,
          body: 'IP '+sourceIp+' already allowed for port '+PORT,
          headers: {
            'Content-Type': 'text/html',
          }
        }
      }
      // Return server error
      console.error(err)
      return {
        statusCode: 500,
        body: JSON.stringify(err)
      }
    }

    // Return success response
    return {
      statusCode: 409,
      body: 'IP '+sourceIp+' allowed for port '+PORT,
      headers: {
        'Content-Type': 'text/html',
      }
    }
  }

  // Log unauthorized request
  console.error('not authorized request from '+sourceIp);

  // Return unauthorized response
  return {
    statusCode: 401,
    body: 'Invalid Authorization Token',
    headers: {
      'Content-Type': 'text/html',
    }
  }
}

As you can see, to verify the origin of the request we opted a solution that pass a certain token as a path parameter. This is because we want the use of this endpoint to be as simple as possible and in this way we can directly open a link similar to this:

https://xxxxxxxxx.execute-api.us-east-1.amazonaws.com/stage/whitelist/yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy

directly in the browser without having to set a particular header (for example X-API-Key using the API Gateway build-in system) that would need, for example, a curl call from the terminal.

The configuration of this token, security group ID, protocol, port and description is done through environment variables, in this way we can deploy the same Lambda code for different environments and infrastructures.

Scheduled cleaning of rules

The default security group has a limit of 60 rules (incrementable up to 1000 via a request to AWS) which may be affected after a few months of system usage. In addition, this automatic system is used exclusively by those who connect using a dynamic IP, this means that the IP will be reassigned to someone else and must therefore be removed from whitelist.

The solution we have adopted is to add another Lambda, this time triggered by a scheduled CloudWatch event, which will clean up the security group rules.

Obviously we do not want the rules added manually for static IPs to be removed as well, otherwise it would cause an inadvertent blocking of third-party integration accesses. To solve this problem we use as discriminant the rule’s description, set through the environment variable ROLE_DESCRIPTION in the Lambda described above. In this way we are going to clean up only the rules added through this automatic system and allowing the DevOps team to manage some rules manually.

The Lambda code is also very simple in this case:

const AWS = require('aws-sdk')
const ec2 = new AWS.EC2({
  apiVersion: '2016-11-15',
  logger: console
});

/**
 * Configs
 */
const RULE_DESCRIPTION = process.env.RULE_DESCRIPTION || 'managed by lambda';
const SECURITY_GROUP_ID = process.env.SECURITY_GROUP_ID
const PROTOCOL = process.env.PROTOCOL || 'tcp'
const PORT = process.env.PORT || 22

/**
 * Lambda handler
 */
module.exports.handler = async () => {

  // Build parameters
  const clearedIpPermission = {
      IpProtocol: PROTOCOL,
      FromPort: PORT,
      ToPort: PORT,
      IpRanges: []
  };

  console.log('cleaning all IPs on security group '+SECURITY_GROUP_ID+'..');

  // describe security group
  const { SecurityGroups } = await ec2.describeSecurityGroups({
    GroupIds: [
      SECURITY_GROUP_ID
    ]
  }).promise()

  if(SecurityGroups.length == 0){
    console.log('security group '+SECURITY_GROUP_ID+' not found');
    return
  }
  
  // list all security group rules
  const ipPermissions = SecurityGroups[0].IpPermissions;
  ipPermissions.forEach(function(ipPermission){
    ipPermission.IpRanges.forEach(function(ipRange){
      // Check if rule has the right description
      if(ipRange.Description === RULE_DESCRIPTION){
        // Save rule to delete
        clearedIpPermission.IpRanges.push({
          CidrIp: ipRange.CidrIp
        })
        console.log('going to remove ', ipRange.CidrIp);
      }
    })
  })

  // Check if there are rules to clean
  if(clearedIpPermission.IpRanges.length == 0){
    console.log('no rules to remove');
    return
  }

  console.log('removing '+clearedIpPermission.IpRanges.length+' rules..');

  // Remove security group rules
  try {
    await ec2.revokeSecurityGroupIngress({
      GroupId: SECURITY_GROUP_ID,
      IpPermissions: [
        clearedIpPermission
      ]
    }).promise()
  } catch (err) {
    console.error(err)
    throw err
  }
}

we directly use the revokeSecurityGroupIngress API by passing the list of ingress rules with a specific description.

Conclusion

This system meets our needs and has a good compromise between security and usability. This system only grants access by a certain IP to the SSH service exposed by the bastion but does not grant access to the machine, whoever wants to connect must still use their own personal SSH key.