When Infrastructure as Code Ends Creating Terraform Providers | Harel Safra | Conf42 PE 2024
Infrastructure as Code simplifies and automates infrastructure management, making it repeatable and efficient.
Hello and welcome to Con 42 Tax Form Engineer 24. I'm Herl, and today I'm going to talk about developing and creating Terraform providers. We'll cover a bit about an Infrastructure as Code (IaC) refresher, where to start when creating a provider, and how to go about creating a provider, including a live demo.
First of all, I'm Har El Saffra, the Data Platform Engineering Team Lead at Risky Five. My team manages all the online databases, everything related to online systems, and access either to end customers or to analytical systems.
Infrastructure as Code is a programmatic definition of infrastructure elements. It allows repeatable, documented processes, eliminating the need to click through UIs or perform the same manual tasks repeatedly. There are two general approaches to IaC: declarative and imperative. The declarative approach involves the user defining what they want to achieve, and then the platform framework does the work for them and provides the infrastructure. The imperative approach defines how they want to create it, iterating over loops and creating code or code-like states that go ahead and create infrastructure elements. We can contrast Terraform and Pulumi on either side.
Terraform providers are the plugins that interface with the infrastructure API. Terraform Core is a framework that knows how to create objects in infrastructure elements but doesn't have any idea about how the infrastructure is created and managed. To interface over that, we create providers. The interface between Core and the infrastructure API is internal, with Core interfacing with a provider over RPC. As of today, there are 4400+ providers, and the list continues to grow. The main thing to remember is that anyone can create more providers. You don't have to be part of the infrastructure team or the product or company at all. For example, when I needed to create a provider for external databases that could be used by my team, I was not part of the database development.
So, where do I start when I want to create a provider? First, understand a bit about the architecture of the process. We have Terraform Core on the left side, which is the executor that lets you run and manage operations. It knows how to create infrastructure, route infrastructure, and calculate deltas between the needed state and the current state. It interfaces over RPC with a Terraform provider, which is a plugin that you will develop. This provider, written in Go, specifically interfaces with a client library. That client library interfaces over a native protocol with the infrastructure. The native protocol could be HTTP calls, gRPC, SQL system calls, or any RPI that the infrastructure knows how to understand and implement.
To create a provider, you need to find the correct API to interface with the infrastructure. There could be old versions and new versions that are easier to work with. If there's a version that has an existing Go client for it, I would advise going for it because it will be easier for you to create the plugin. Obviously, as I mentioned, the provider is written in Go, so you need to know and understand Go. Go is a compiled and high-level programming language. You don't need to have a deep understanding of it, but you should get familiar with it.
Choose the API version with an existing client to simplify your plugin development.
To implement the API you need, first, you must find the correct API to interface with the infrastructure. There could be old versions and new versions that are easier to work with. However, if there's a version that has an existing Go client for it, I would advise going for it because it will be easier for you to create the plugin with that.
Obviously, as I mentioned, the provider is Go, so you need to know and understand Go. Go is a compiled and high-level programming language. You don't need to have a specific deep understanding of it; you can use Google Chat GPT to go along, but you need to understand the constructs, control structures, and anything like that. There's a good step-by-step tutorial that I used under go.d tour. It will help you go step by step learning Go, starting from basic primitives, control loops, infrastructure, and interface.
I found Go to be a simple language to understand and learn, especially if you have knowledge of other programming languages. It's compiled, which I like because the compiler finds errors that you don't bump into during runtime. One caveat that I found is that there are no exceptions, meaning you need to specifically check for error conditions from function calls; otherwise, your code will just panic. Remember to check return values—the error return value is your friend.
HashiCorp documentation is the thing to use and it's recommended. It's located in their development portal, and you can also scan the QR code to find that. Read the docs again, and they will give you a basic understanding of how strings plug into one another. After you've learned Go and read the documentation from the HashiCorp documentation, you can go ahead and create a provider.
I was using a demo to make this a bit more real. This demo is a plugin that manages lines in a text file—simple lines and a simple text file. All files are managed in a single path. There's a file resource that has a file name and the lines array. The file API was provided for you by me and is limited. It doesn't have a lot of primitives; it could read lines, write lines, and count the number of lines in the file, but not a lot more than that. You can see the resource definition on the left; it has a file, file one, and two lines, line one and line two, which maps into a file that has two text lines.
You can grab the code under my GitHub repo H7 terone provider fin data. You can also scan the QR code to get there. The Terraform plugins framework is the recommended way to create new providers. There are various ways—two or three different ways—and due to an API specification change, the new way is the plugin framework. It allows you to manage your business logic and focus on your business logic while the plugin framework does the hard lifting of connecting to Terraform Core over RPC.
You start working on that by cloning the Terraform providers scaffold repo to your private repository and then customizing it from there. When working with Terraform, there are four basic operations that Terraform Core needs to be able to run over infrastructure: they provision infrastructure and amend the Terraform state accordingly. The create operation obviously creates the resource; it could create a file in our case, an EC2 server, or a user in a managed SaaS database. The read operation reads the current infrastructure state. The update operation changes the infrastructure state.
Master Terraform by understanding its core operations: create, read, update, and delete.
When cloning the Terraform providers repository to your private repository, you can customize it from there. When working with Terraform, there are four basic operations that Terraform Core needs to be able to run over infrastructure. They provision infrastructure and amend the Terraform state accordingly. The create operation obviously creates the resource; it could create a file, an EC2 server, or a user in a managed SaaS database. The read operation reads the current infrastructure state. The update operation changes attributes that can be changed according to the API. Obviously, not all attributes can be changed. For example, if you want to recreate an EC2 server, some things can be updated like the SSH key, but some attributes cannot be updated. The update API handles these changes. The last operation is the delete operation, which removes the resources. Terraform Core also uses this for recreate operations in case a user wants to change attributes that cannot be changed without destroying. Terraform Core will go ahead and delete and recreate the resource by using destroy and create again.
If you look over at the code, you can see that this is for cloning and a few things that I mentioned earlier. The file API that I provided for you has a few basic operations: read, count the lines, remove files, and remove line files. That's something that I added for comparison and examples. If you look at the file resource, you can see that each of the primitives that I mentioned earlier has a method associated with it. We have a delete method, an update method, a read method, and a create method, among others that we will cover a bit later on. If we expand the create method, for example, we can see that it has code that interfaces with Terraform Core. It basically copies the request from Terraform Core into a local variable. Then there is logic that performs the operation and infrastructure, in this case, creating a file and writing lines into it. It then returns the value back to Terraform Core to allow it to know what happened and amend the state accordingly.
Each of the resources and data has schemas and attributes. Schemas and attributes are the mapping between Terraform configuration blocks and the code. They define what parameters are needed to create infrastructure and change its values. Some of these parameters can be mandatory, while others can be optional. These parameters are necessary to create the infrastructure itself. Schemas have attributes that define specific data elements, and each attribute has a type. A type could be a primitive, an integer, a string, or a complex data type like a map, an object, or a list. Each attribute also has properties such as description, optional, sensitive, and other properties. They can also have optional validators to check that the user-supplied values match what the infrastructure provider requires.
If you take a look at the code, you can see that this is all implemented in a schema function. The file has two attributes. First of all, the file itself has a description: file data resource. It has two attributes: one of them is a file name, which has a description of "file name". It is required and has a validator that forces the file to match a specific regular expression.
Validate user inputs with schema functions to ensure they match infrastructure requirements.
The code implementation involves optional validators to ensure that the user-supplied values match what the infrastructure provider requires. If you take a look at the code, you can see that this is all implemented in a schema function. The file has two attributes. First, the file itself has a description: file data resource. It has two attributes: one of them is a file name, which has a description of "file name". It's required and has a validator that forces the file to match a specific regular expression. Additionally, it has an attribute called lines, which is a list attribute, and each one of those is a string value. In this specific case, it also has a validator that forces the line function to have at least two lines, just as an example.
You should know that descriptions under the attributes are used because they are copied over to the automatically created documentation. This documentation essentially copies the description and filing, and the element type here, and then you can see this documentation later on copied into the Terraform registry for your end users. The plugin schema types are not native Go types; they have additional methods and functionality to handle null values and unknown values. For example, in N64 and other primitives, there is an isNull method that returns if it's null or not. Primitives use the value type method; for instance, the N64 has a method called valueN64 that returns a native Go int64. Collections are converted into Go types with as methods. For example, with a list, you can grab the elements of a list as native Go types with elementsAs and then cast them into a slice or array of strings.
If you go back to the code and look at the create function, you can see that the full name of the file is created by appending the base path for the provider with the file name. The file name is a framework string type, but we grab the Go length value with valString, which turns it into a native string that we can use to append into values. After you have created the provider and are quite happy with it, you will want to run it to see that it works. First of all, you don't have to publish the Terraform provider into the Terraform registry to run it; you can run it locally. You can do that by using the dev overrides inside the configuration file and then send the TFCLI file to that file. For example, you can see that with the CLI configuration file, there is a Dev override that maps the registry address into a local path on my Mac.
You can use log-based debugging for simple cases, but you can also use debugger-based debugging, which is far more powerful for complex ones. You do that by running the main method of your provider with debug=true. It will output in the environment variable, and you copy it to wherever you want to run it, and then you run the Terraform action, and the action will break the debug. Let's see how that works. First of all, I have my code here. Let's look at the Run configuration and see that everything works correctly. We have a debug, and you can see that the program argument has debug=true. I set a breakpoint, for example, let's say I want to debug the read operation, so I set the breakpoint here and then I run the...
Debugging Terraform providers is a breeze when you set breakpoints and use the debugger to step through your code!
By running the main method of your provider with d debug equals true, it will output in V dver. You copy it to wherever you want to run and then you run terraform action. The action will break the debug. Let's see how that works. First of all, I have my code here. Let's look at the Run configuration and see that everything works correctly. We have a debug, and you can see that the program agreement has death debug equals true. I set a breakpoint, for example, let's say I want to debug the read operation, so I set the breakpoint here and then I run the program. It will compile and then output an environment variable which I copy and then paste here as it is.
Now, this directory has a configuration for terap. It has a provider block, a provider file that defines the base path, which is the basa directory. It also has a resources file that defines two file resources: file one and file two. You can see that each file has a few lines, and if we cap file one, for example, you can see that it has two lines. As I mentioned earlier, I exported the TF fre attach providers enable. Now, if I run terraform plan, it will start, and then it will pop up the debugger. You can see that the terraform run itself is waiting for the debugger output. You can use the debugger to scroll through your code like everywhere else. For example, you can see that the full name created is a read operation for file two. If we resume the program again, it will continue running and then end in refresh state. If you don't need that, you can see that it refreshes the state and the relation. In this case, there's no change in the file, so there are no changes in the infrastructure. When you're done with that, you obviously want to stop the debugger and then go back to the presentation.
After you debugged your code, you're happy with your test, automatic test, and acceptance test for your resources and data sources. This allows you both to check that the code is working correctly and later interface into the GitHub actions. The state is already checked basically in this test. If you want to create and test operations by yourself on the infrastructure stuff, you need to add them to the acceptance test. You can run the test manually with make test AC. The way it happens is to create a resource uncore test.go file in the same directory, and the make will grab them. If we go back to the code and close the debugger menu, you can see that, for example, the file resource has a file resource test.go. This file has functions, each function is a test that has pre-check steps for the test.
As I said, the test state checks in Bally. For example, in this case, it comes with a function that we'll see a bit later. It creates a file F with two lines, one and two, and checks the attribute to ensure that the file name is file one. The second step is to update file one with lines two and three instead of one and two and check that the first line is equal to two. Both of these functions use the set function test x five config, but it could be whatever you want that just returns a terraform configuration file. Basically, we have a fer stanza and res stanza, and you can see that the values are appended here to make creating configuration files easier instead of writing them explicitly again and again.
When we're done with that, you can run the acceptance test, and in this case, they pass, which is nice. You've created your code, debugged it, created acceptance tests, and documentation. You're all fine and happy, and you want to publish it into the terraform registry. The first step is to create a GPG key that you will use for validation. The terraform registry will use this to validate that the code is indeed yours. Set the G secrets GPG private key head password accordingly. Create a git tag named V and version it, for example, V0.1.0 or V2.3. Make sure to follow semantic versioning. This tag drives the GitHub action included with the terraform provider plugin framework. That action builds all the executables and also pushes a webhook to terraform registry to allow you to grab the new versions.
The next step is to log into the terraform registry and add the repo. Once you've done that, authenticate back to GitHub. It will create the webhooks that will push new updates tagged into the terraform registry. Wait a bit; it takes a few minutes for terraform to grab the changes from GitHub. Then you can see your provider published into terraform for other people to use. Everything published is available to the public, so be aware not to publish any proprietary code or anything secret.
To wrap it up, my team and I started from manual management of this specific database. We have used terraform extensively for other cases, but for this specific database, we had manual management scripts and Confluence. I had to learn the terraform framework, created a provider, and released a few more features called version z. I published the provider, my team used it, and I hope that other people around the world use it too because it makes managing a database easy. Thank you all for your time. I'm Har Safer. You can find my contact details and GitHub profile below. I welcome any questions. If you have anything, reach out to me, and I'll be glad to help with anything you need. Thank you for your time and for being here with me.