Hybrid SaaS balances a SaaS control plane with customer managed compute. It offers a solution for security conscious customers to take advantage of some of the ergonomics of SaaS, without outsourcing as much of the risk of the system. This has become particularly popular with the rising concerns around vendor compromise. There is a standard security architecture emerging for these systems.

This talk will break down the controls and compromises in a real world deployment of a Hybrid SaaS architecture for Terraform Automation. It will both discuss the standard security model of hybrid SaaS, and also how to maintain a high security bar for Terraform Automation without accepting the developer experience compromises of other solutions.

Previous talks and research, including the excellent Attacking and Defending Infrastructure with Terraform: How we got admin across cloud environments at BSidesSF2022, focus on the security risks in Terraform Automation. However, that talk and the general guidance tends to rely on limiting who can interact with your terraform automation - for example by limiting who can run a Plan - as a core security control.

At Figma, we’ve deployed a system that leverages compensating controls to mitigate the risks without erroding developer experience. This is tightly coupled to our use of Hybrid SaaS.

The talk will specifically cover Terraform Automation security model (using Spacelift), and will pull in our experience securing Buildkite - another Hybrid SaaS system - to show how many of the lessons can be generalized across Hybrid SaaS.

While the talk will mention these specific vendors, the goal of their inclusion is to give specific context on the available risks, controls, and limitations provided as Hybrid SaaS. The talk will explicitly call out that alternative vendors for Terraform Automation exists, and controls - as a benefit of hybrid SaaS - should not be specific to Spacelift customers but generally applicable.

Introduction - 5m

• Refresher on Terraform Architecture (ex. https://spacelift.io/blog/terraform-architecture)
  • Specific mention of third party modules and providers

Attacking Terraform Automation - <10m

• Review of Terraform Automation (including mention of various vendor and OSS solutions)
• Review prior art on attacking Terraform Automation (with “demos”/recordings/visualizations of attacks)
  • Malicious providers and modules
    • https://about.gitlab.com/blog/2022/06/01/terraform-as-part-of-software-supply-chain-part1-modules-and-providers/
    • https://sprocketfox.io/xssfox/2022/02/09/terraformsupply/
  • RCE in Terraform Plan / “Apply-in-Plan” * https://cloud.hacktricks.xyz/pentesting-ci-cd/terraform-security#rce-in-terraform
    • https://youtu.be/yt-3ndhMMn8?si=wC1Rt1F8NEXLzUEb&t=1006
  • Secrets in State Files
  • “Task Runs” * https://developer.hashicorp.com/terraform/cloud-docs/architectural-details/security-model#what-isn-t-part-of-the-threat-model

Hybrid SaaS - <10m

• SaaS control plane, customer owned compute
• Pros / Cons
  • Increased customer ownership/control of security is both
  • “can we do security better than the vendor”
  • “how sensitive is this system to risks with multi-tenancy”
  • Increased management overhead -> “potential worst of both worlds in paying for SaaS and also having to manage compute”
• Examples at Figma
  • Buildkite
  • Spacelift
• Security model
  • specifically: highlighting signed payloads as a core element
  • https://buildkite.com/docs/agent/v3/securing#signing-pipelines
  • https://docs.spacelift.io/concepts/run/#zero-trust-model
  • discuss “how to decrease trust in the control plane” as a driving motivation
    • Secrets management
    • Prioritize putting security in the worker, not the control plane. The latter can be ignored on compromise
    • IAM -> Figma’s CI Credential Exchange for Buildkite, OIDC-based assumption for Spacelift

Securing Spacelift @ Figma

• Four threats
  1. “proposed runs” -> if you can put up a PR, you can run a Plan
  2. compromise of engineer session on control plane
  3. compromise of admin session on control plane, or Spacelift itself
  4. “tracked runs” -> if you can get code in master, you can run an Apply
• Threat 1 is “outside the security model” for Terraform Cloud, and is often framed as untenable (i.e Mike’s talk)
  • What do you do if you refuse to compromise on the DevEx of allowing anyone to see a Plan of their changes
• Controls
  • Lean on Github Approvals + proprietary Commit Integrity (signing, validation)
  • Private Worker Pool -> Spacelift has no direct permissions in Figma’s AWS infrastructure
    • Account level isolation of these workers
  • Worker side validation of all commits sent to workers via Github APIs
  • Permission Segmentation between Plans and Applies
    • Sandboxing of “runs” as a core enabling architecture
    • Next Step –> roll our own Stack level isolation using OIDC
  • Semgrep for static analysis to attempt to detect known RCE patterns, with alerts on violation
    • Next Step –> Migrate to REGO with spacelift-launcher --run-initialization-policy
  • Spacelift Policies - role based approval
    • ergonomics -> Auto-Approval allowed where safe!
    • opal -> JIT access to approval groups
  • Securing the configuration definition with Commit Integrity
    • note: figma doesn’t enforce code ownership, reference devex tradeoffs in Github “stale approvals”
  • RBAC in control plane to limit risk of engineer compromise
  • Disable “Task Runs” (RCE-as-a-service) both control plane and worker side
  • JIT authorization + detective controls for Admin actions
  • Vendor all terraform modules, semgrep to warn at PR time
  • Minimize providers per-state to minimize risk of provider takeover
  • Use a mirror for all Terraform providers
  • Do not store secrets in state. Implemented secrets scanning for terraform state
  • Pin “worker image” in case of Spacelift compromise
    • TF_LOG -> Check in pre-init, unset if it’s a dangerous value

Panther rules

• Stack modified by unexpected principals
• New API keys // Actions from API keys
• New AWS integrations // task runs

High Integrity

• runner image from our ECR repo
• only master tracked runs, only our main repo
• must have run static analysis in pre-init
• ban task runs
• distinct worker pool
• more aggressive alerting on control plan logs

1. This talk will heavily leverage content from Beyond the AWS Security Maturity Roadmap, which I presented at fwd:cloudsec 2023. I’ll be adding an additonal half-dozen controls/program elements that were cut for time in the 30m version, and expanding the amount of context setting to suit a slightly broader audience. This will still be an “advanced” talk that focus on mature programs, but as a survey of those programs still offers interest and accessibility for audience members who aren’t working on cloud security directly.
2. The talk leans practical and actionable, focusing on technologies and controls that can be implemented today, in real businesses, at real scale.

Rough outline:

Context: 12m

• Biases on “the sort of cloud security program” I’m talking about: engineering oriented security program, zero trust, selling a software product
• The baseline - a quick overview of what is already assumed based on Scott’s Cloud Security Maturity roadmap
• The Netflix influence
  • “Netflix exists in order to spite the gods, copy them not”
  • To break it down: many of the solutions shared are built with constraints and opportunities that make it not apply to you
    • Netflix: predates many managed services, oss offerings, and cloudsec products. they have scaled past the utility of many off-the-shelf solutions. they have enormous data gravity and have to hack around their account architecture.
• When I joined Figma, the pitch was - “we’ve shipped the roadmap, help us figure out what’s next.” As an example of where the program was at - refer to MaxB’s 2021 fwd:cloudsec talk
• Build v. Buy
  • The type of cloudsec team with engineering capabilities has this option
    • In many cases, just a few years ago the only option would be to Buy
  • Framework from Sabry Tozin
    1. Are we solving a problem unique to our company?
    2. Are we solving a problem at a scale unique to company?
    3. Is the cost and effort of integrating an off-the-shelf solution so large that we may as well build one?
    4. What are the purchasing/ongoing license costs of the product in comparison to building it ourselves? 1. ref: https://boringappsec.substack.com/p/edition-9-a-build-vs-buy-framework
       • Bad reasons: Buying to de-risk, Building for vanity
       • be pragmatic about security risk in involving a 3rd party in this solution

Meat: 25m

We’ll talk about the problem space as you scale and give specific open source and popular-in-peer-companies commercial solutions to:

• Asset inventory / continuous compliance
• Secrets Management
• Secure IAC modules
• SSH replacement
• Least Privilege / IAM
• Account management, vending, and sandbox accounts
• DFIR
• Automated remediation
• Runtime Security
• Endpoint Monitoring
• Egress / Perimeter (data, network)
• Honeytokens
• etc.

This talk will respect the fwd:cloudsec audience, and take Scott’s roadmap as lingua franca.

Outline:

Context: 6m

• Biases on “the sort of cloud security program” I’m talking about: engineering oriented security program, zero trust, selling a software product
• The Netflix influence
  • “Netflix exists in order to spite the gods, copy them not”
  • To break it down: many of the solutions shared are built with constraints and opportunities that make it not apply to you
    • Netflix: predates many managed services, oss offerings, and cloudsec products. they have scaled past the utility of many off-the-shelf solutions. they have enormous data gravity and have to hack around their account architecture.
• When I joined Figma, the pitch was - “we’ve shipped the roadmap, help us figure out what’s next.” As an example of where the program was at - refer to MaxB’s 2021 fwd:cloudsec talk
• Build v. Buy
  • The type of cloudsec team with engineering capabilities has this option
    • In many cases, just a few years ago the only option would be to Buy
  • Framework from Sabry Tozin
    1. Are we solving a problem unique to our company?
    2. Are we solving a problem at a scale unique to company?
    3. Is the cost and effort of integrating an off-the-shelf solution so large that we may as well build one?
    4. What are the purchasing/ongoing license costs of the product in comparison to building it ourselves? 1. ref: https://boringappsec.substack.com/p/edition-9-a-build-vs-buy-framework
       • Bad reasons: Buying to de-risk, Building for vanity
       • be pragmatic about security risk in involving a 3rd party in this solution

Meat: 14m

We’ll talk about the problem space as you scale and give specific open source and popular-in-peer-companies commercial solutions to:

• Asset inventory / continuous compliance
• Secrets Management
• Secure IAC modules
• SSH replacement
• Least Privilege / IAM
• Account management, vending, and sandbox accounts
• DFIR
• Automated remediation
• Runtime Security
• Endpoint Monitoring
• Egress / Perimeter (data, network)
• Honeytokens

This panel will gather several of the participants of https://tldrsec.com/guides/staffeng-security and build on that work. We hope to continue the discussion of Staff+ Engineering in Security and offer an opportunity for the BSidesSF community to have a conversation with a broad set of security engineers who have reached these roles.

outline

We’ll start with a quick introduction,which means each panelist gets 90 seconds to pitch themselves. We’ll open up with some crowdsourced and pre-seeded questions to get the discussion going, covering topics like (example questions):

• What’s a week like as a Staff+ engineer?
• What are the hardest parts of being a Staff+ engineer?
• What should you do if you feel “stuck” at Senior?
• What about your Staff+ journey or role is specific to Security?
• Is there an IC/Manager Pendulum in Security?
• What comes after Staff+ for you?

After that we’ll open the floor to questions from the conference participants. We believe this format will:

• Allow panelists to set shared context on Staff+ Engineering roles in security
• Share some of the key takeaways and themes from https://tldrsec.com/guides/staffeng-security
• Provide plenty of opportunity for conference participants to engage in a BSidesSF specific discussion

Elevator Pitch

You can’t buy security, but vendors play a key role in effective security programs. This talk will provide a comprehensive guide to buying and getting value, based on experiences on both sides of the marketplace, a comprehensive literature review, and a survey of clients and vendors of all stripes.

Talk Description

A detailed outline and the key takeaways for your presentation. Tell us more details about your presentation and why people will want to attend. These details will help us make an informed decision about your submission.

• You can’t buy security, but security services vendors play a key role in effective security programs. They can offer much needed staff in the current tight market for experienced talent and bring expertise that is not cost-effective to maintain in-house. Vendors are also engaged for the growing requirements of compliance and standards. A third-party can provide objective feedback and bring broad industry context to bear.
• This talk will provide a comprehensive guide to buying and getting value from security services. The content comes from experience on both sides of this marketplace, a comprehensive literature review, and a survey of clients and vendors of all stripes. We’ll discuss the landscape of vendors and engagements, and go through a structured approach to buying security. We’ll step through deciding you need security services, contracting, processing the results and moving your security program forward.
• Shopping for and deriving value from these vendors is becoming a core competency for security professionals. This is an area many organizations struggle due to information asymmetry, difficulty of assessing performance and quality, and infrequent or limited awareness of the breadth and quality within the market. Combined with misleading sales tactics and a lack of standardization on delivery, this overwhelming marketplace, now with even more alternatives to traditional single-vendor assessments - buying a decent security engagement is harder than ever.

Notes

Note: I’m able to present this as a 25 minute talk instead. It’d be tight with the target breadth, and so will likely focus heavily on survey results instead of comprehensive procedure.

Why this?

To some extent, this topic sounds like well tread ground, but there are a number of distinctive elements that make this worth sharing: 1. It’s vendor agnostic: While I have a background as a Security Consultant - I am explicitly disinterested in discussing specific vendors, even implicitly. Roughly ⅔ of the 50+ sources reviewed for this material came from vendors attempting to sell their services. 2. It’s built on a comprehensive literature review: I’ve read and compiled a massive amount of similar work, all of it flawed in various ways. This includes everything from vendor and client blogs, to standards attempts (PTES), analyst works (Gartner), and authoritative guides (CREST, NCSC). This talk functionally is a superset of all the best guidance available today.
3. It includes the data from a survey: I’ve been working with Clint Gibler (tldrsec.com) to develop this content, which will include a survey. I’ve already received some great content from everyone from Directors at FAANG to first security hires to solo consultants and folks from the big security services vendors. Taking advantage of TL;DR Sec’s community of 8,000+, I’m confident there will be some excellent takeaways to share. If accepted to BSidesSF, I will hold off publishing the longform written version in conjunction with my talk.

Why me?

I’ve spoken previously at multiple BSides events (CT 19+20, Boston 20) as well as DEFCON Cloud Village 2021. I’ve supported, sold, and delivered dozens of security assessments, and then went in house and bought services as a Staff Eng. at a scale-up startup. I create comprehensive and objective content that is deeply seated in prior art - such as: https://tldrsec.com/blog/cloud-security-orienteering/ and https://research.nccgroup.com/2020/04/24/the-extended-aws-security-ramp-up-guide/

Outline

An outline of the non-survey content:

1. Types of Security Services
2. Common motivations, and impact on vendor selection
   1. Compliance, Sales, Investment or M&A, internal attestation, post-breach, risk reduction
3. Types of Security Services Vendors
   1. Global/enterprise consulting, cyber security services, boutique (or regional), specialty, sole practitioner (researcher/bug hunter), low cost, MSSP, VAR
4. How to find potential vendors
   1. Network recommendations, follow-the-leader (industry endorsements), research, conference speakers, published research, public reports, compliance approved, assessment standards work, prominent staff, certifications, analyst recommendations
5. Client-side scoping and requirements
6. Requesting and reviewing proposals
   1. Running an RFP: what to ask vendors, what to expect
   2. Vendor-side scoping
7. Contracting
   1. Quotes, negotiation, vetting, rules of engagement, common clauses, change orders
8. Pre-assessment preparation
   1. Technical, onboarding, logistics
9. After the assessment
   1. The readout, how to read a report, ingesting the results to the organization
10. What comes next

A proposal on expected survey content:

1. High-value topics for compiled data include:
   1. Pricing (bucketed, day-rate focused)
   2. Contracting (key terms)
   3. Negotiating
   4. Scoping
   5. Consulting Program Management
2. “Pull quotes”/specific interesting responses - ex from the current preliminary sample:
   1. Notes on a bakeoff held between consultancies by a high-profile buyer
   2. What a former consultant has found buyers value that he didn’t appreciate
   3. Takeaways from selling millions in services to enterprises
   4. The perspective of a small VAR selling

Key takeaways will be focused on making audience members aware of the breadth of the industry, and understand how to introspect on their needs, use that to generate a list of potential vendors, and them make an informed decision on contracting. For those who have never bought security services it will take them from 0-60, and for professionals who have bought engagements it will interrogate their assumptions on the quality of their procurement.

Intro

• About Me
• Undifferentiated work
  • What is it? What does it look like?
    • ex. Dumb Security Questionnaires (https://latacora.micro.blog/its-weird-to/)
  • Differentiated: your risks, your product, your industry, your team

Operating principles

• this is a maturity shortcut
• there are a lot of bad resources, so we must focus on separating the signal from the noise
• Only bring things into your program that you understand
  • corollary, only bring things in that you believe in and can evangelize

• Identify a need → fill it, don’t do a solution in search of a problem

• Security Training and Evangelism:
  • https://sudo.pagerduty.com/: Training presentations for everyone, and for engineers
  • https://portswigger.net/web-security: Free, online web security training
  • https://github.com/c0rdis/security-champions-playbook: How to set up a security champions program
  • https://cams.mit.edu/wp-content/uploads/Verizon-Media-CyberCulture-Paper.pdf: Verizon Media’s Proactive Engagement team six things managers can do to drive cybersecure behaviors
• Incident management
  • https://response.pagerduty.com/: preparing for an incident, and what to do during and after the incident
  • https://codeascraft.com/2012/05/22/blameless-postmortems/
  • https://codeascraft.com/2016/11/17/debriefing-facilitation-guide/
    • Etsy’s pioneering guide to blameless postmortems
• Risk Assessment
  • https://infosec.mozilla.org/guidelines/risk/rapid_risk_assessment.html: A quick risk-focused approach to analyzing and assessing services
  • https://partner-security.withgoogle.com/docs/security_review_process.html: A boilerplate Vendor Security Assessment (VSA) process and questionnaire
  • https://github.com/magoo/minimalist-risk-management: A minimally viable risk management program
• Vulnerability Reporting
  • https://dropbox.tech/security/protecting-security-researchers
  • https://github.com/disclose/dioterms + https://policymaker.disclose.io/policymaker/introduction/ + https://www.ncsc.gov.uk/information/vulnerability-disclosure-toolkit
  • https://bugbountycoi.org/2021/05/04/chapter-1-what-is-a-bug-bounty-program/ + https://www.lutasecurity.com/vcmm
• Compliance
  • Controls to Standards mapping: https://blog.adobe.com/en/publish/2020/11/19/common-controls-framework-ccf-by-adobe-v4-0-now-available.html
  • https://gdprchecklist.io/
  • https://github.com/JupiterOne/security-policy-templates
• Hiring
  • https://danielmiessler.com/study/infosec_interview_questions/
  • https://github.com/tadwhitaker/Security_Engineer_Interview_Questions/blob/master/security-interview-questions.md
• Everything from Gitlab: https://about.gitlab.com/handbook/security
• 18F: https://engineering.18f.gov/security
• https://www.bishopfox.com/cybersecurity-style-guide/
• https://github.com/basecamp/policies/blob/master/security/Basecamp Security Overview.pdf
• Penetration testing: https://partner-security.withgoogle.com/docs/pentest_guidelines
• Checklists!
  • Pick a tool, technology, or framework
  • someone has made a good checklist of common issues
  • this is a wheel I’ve seen re-invented way too many times
  • watch our for vendor blogs, favor individuals and pentesting firms
  • https://github.com/tprynn/web-methodology/wiki
  • CIS Benchmarks are a fallback
  • also: https://www.ncsc.gov.uk/collection/device-security-guidance

Introduction:

1. To me
2. To AWS Security
   1. “The cloud”
   2. Shared responsibility model
3. To the topic - Breaches of AWS Environments

A quick hat tip to prior art:

1. SANS Cloud Security Summit
   1. Lessons Learned from Cloud Security Incidents, Past and Present
      1. https://www.youtube.com/watch?v=wUYKq1F0Qnk
   2. Cloud Breaches: Case Studies, Best Practices, and Pitfalls
      1. https://www.youtube.com/watch?v=rtEjI_5TPdw
2. fwd:cloudsec
   1. “The Usual Suspects” - James Condon
3. F5 Is the Cloud Safe? Part 2: Breach Highlights for the Past 3 Years (https://www.f5.com/labs/articles/threat-intelligence/is-the-cloud-safe–part-2–breach-highlights-for-the-past-3-years)

Case Studies: (for each I’ll break down the breach, and identify the root cause)

1. The simple examples - exposed S3 buckets, managed Elasticsearch, etc.
2. The one we all heard about - CapitalOne
3. Code Spaces
4. DNC Hack by the GRU
5. LA Times
6. OneLogin
7. Uber
8. Impreva
9. Tesla
10. JW Player
11. TeamTNT botnet
12. Cryptomining AMI
13. Mandiant talk example
14. Expel.io examples
15. whatever happens between now and the conference?

Root Causes: (I’ll correlate the breaches, and establish a set of common root causes, this can be compared to both https://disruptops.com/stop-todays-top-10-cloud-attack-killchains/ and https://attack.mitre.org/matrices/enterprise/cloud/ “Inital Access”. For each, we’ll do a 10,000ft look at best practices for prevention)

Outline:

I’m not sure whether the goal is to allow for questions, and if so whether that is included in the talk time or appended afterwards. This rough outline provides the flexibility for either case.

10 Minutes

• Introduction
• To me
• To the topic
• To “the Cloud”
• To AWS
• AWS Shared Responsibility Model
• Key AWS Background (terms and technologies)
• Networking in AWS
• VPCs, EC2, security groups, NACLs
• IAM
• IAM Principal Types (Users/Roles/root)
• Credentials (Passwords/Access Keys)
• Policies
• Policy Evaluation

15-20 Minutes

• “Easy”
  • Single account best practices
  • Single account security services
  • Common account compromise footholds
  • Single account auditing
  • how to use it for service hardening and best practices
  • Turnkey encryption

15-20 Minutes

• “Hard”
  • Organizational Architecture
  • Security at scale
  • Encryption
  • Least Privilege
  • tips for IAM creation and auditing
  • Access analyzer/Access advisor
  • Logging/Monitoring/Alerting
  • Preparing for Incident Response
  • Visibility and multi-account auditing
  • Antiope
  • JIRA/Salesforce integrated OSS
  • Security Hub
  • Governance
  • IaC
  • AWS Config
  • Organizations SCPs
  • Systems Manager
  • Automatic Remediation

I’ve previously presented on AWS Security and Self-Auditing at BSidesCT 2019, and gave a 4-hour workshop on AWS Security at BASC2019. Slides for both are available at https://speakerdeck.com/ramimac

I have not presented this specific presentation, however it has overlap due to its nature as a discussion of AWS Security.

I’ve tried to create a broad presentation that will have appeal to both those new to the topic as well as those who may already be literate in AWS security. The goal is to work from first principles, but to quickly ramp up to also address the bleeding edge and big picture problems.

This talk will be of interest to those less familiar with AWS Security due to the high level coverage, inclusion of easy wins, and peek into the complexities that come with enterprise scale. For those who are already involved somehow in AWS Security, this can be used for both Gap Analysis, and could assist in identifying roadmaps for future improvements.

My background as a security consultant gives me a valuable perspective through exposure to numerous environments of all sizes, configurations, and maturity.