Fluentd

Overview

Since you deploy microservices on Clever Cloud, you may need some data pipes between your services to:

• collect data from your PostgreSQL to create Elasticsearch indexes for your website search engine
• collect application logs to analyze them with Elasticsearch, then archive them with S3
• collect Apache access logs to analyze them in MongoDB
• extract data from the database of your PHP/MySQL application to transform then load them in your other node.js/PostgreSQL application
• and many more …

Fluentd is an open source data collector written in Ruby, which lets you unify the data collection and consumption for a better use and understanding of data.

Create an application on Clever Cloud

With the web console

Refer to Quickstart for more details on application creation via the console.

With the Clever Tools CLI

1. Make sure you have clever-tools installed locally or follow our CLI getting started guide.
2. In your code folder, do clever create --type <type> <app-name> --region <zone> --org <org> where :
   1. type is the type of technology you rely on
   2. app-name the name you want for your application,
   3. zone deployment zone (par for Paris and mtl for Montreal)
   4. org the organization ID the application will be created under.

Refer to clever create for more details on application creation with Clever Tools.

Setting up environment variables on Clever Cloud

With the Clever Cloud console

1. Go to the Clever Cloud console, and find the app you want to fine tune under it’s organization.
2. Find the Environment variables menu and select it.
3. In this menu, you will see a form with VARIABLE_NAME and variable value fields.
4. Fill them with the desired values then select Add.
5. Don’t forget to “Update Changes” at the end of the menu.

With the Clever Tools CLI

1. Make sure you have clever-tools installed locally. Refer to our CLI getting started.
2. In your code folder, do clever env set <variable-name> <variable-value>

Refer to environment variables reference for more details on available environment variables on Clever Cloud.

You can of course create custom ones with the interface we just demonstrated, they will be available for your application.

Configure your Fluentd + Docker application

Mandatory configuration

To follow this tutorial, you will need:

• Ruby >= 2.4.4 (w/ Rubygems)
• Bundler
• Docker
• Git
• curl
• a Ruby versions manager

My application does not exists already

Create a fluentd application locally

mkdir myFluentd
cd myFluentd
touch Gemfile Dockerfile go.sh td-agent.conf
chmod +x go.sh

Inside Gemfile put the following:

source 'https://rubygems.org'

ruby '2.4.4'

gem 'fluentd'
gem 'fluent-plugin-td'

Then run bundler to install dependencies and generate your Gemfile.lock

bundle install

Clever Cloud needs that your application answers on requests made on 0.0.0.0:8080, we’ll use a PORT environment variable for local test purposes (this variable is automatically setup on each application). Inside td-agent.conf put the following:

<source>
  @type monitor_agent
  bind 0.0.0.0
  port "#{ENV['PORT']}"
</source>

Inside go.sh put the following:

#!/bin/sh

bundle exec fluentd --use-v1-config -c td-agent.conf

echo "🌍 Fluentd server started"

Test locally

Start you service

PORT=9292 ./go.sh

Verify that it responds to requests

curl 0.0.0.0:9292

You can now read My application already exists

Fine tune you application

You can update your configuration with all inputs, filters and outputs you need or check for a community based plugin.

My application already exists

Prepare your application for deployment

Create a Dockerfile at the root of your project and put inside the following (assuming your start script is in go.sh):

FROM ruby:2.4.4
EXPOSE 8080
COPY Gemfile Gemfile.lock td-agent.conf go.sh ./

RUN bundle config --global frozen 1
RUN bundle install
RUN chmod +x go.sh

CMD [ "/go.sh" ]

Environment injection

Clever Cloud injects environment variables from your application settings as mentioned in setting up environment variables and is also injecting in your application production environment, those from your linked add-ons.

Custom build configurations

On Clever Cloud you can define some build configuration: like the app folder to deploy or the path to validate your application deployment is ready To do that follow the documentation here and add the environement variable you need.

Git Deployment on Clever Cloud

You need Git on your computer to deploy via this tool. Here is the official website of Git to get more information: git-scm.com

Setting up your remotes

1. The “Information” page of your app gives you your Git deployment URL, it looks like this:

   1. git+ssh://git@push.clever-cloud.com/<your_app_id>.git
   2. Copy it in your clipboard

2. Locally, under your code folder, type in git init to set up a new git repository or skip this step if you already have one

3. Add the deploy URL with git remote add <name> <your-git-deployment-url>

4. Add your files via git add <files path> and commit them via git commit -m <your commit message>

5. Now push your application on Clever Cloud with git push <name> master

Refer to git deployments for more details.

Linking a database or any other add-on to your application

By linking an application to an add-on, the application has the add-on environment variables in its own environment by default.

On add-on creation

Many add-ons do exist on Clever Cloud: refer to the full list and check add-ons dedicated pages for full instructions.

During add-on creation, an Applications screen appears, with a list of your applications. You can toggle the button to Link and click next. If you finish the process of add-on creation, the application is automatically linked to it.

Add-on already exists

In the Clever Cloud console, under the Service Dependencies menu of your application, you can use the Link add-ons dropdown menu to select the name of the add-on you want to link and use the add button to finish the process.

You can also link another application from the same page in the Clever Cloud console, using the Link applications dropdown menu.

More configuration

Need more configuration? To run a script at the end of your deployment? To add your private SSH key to access private dependencies?

Go check the Common configuration page.

You may want to have an advanced usage of your application, in which case we recommend you to read the Administrate documentation section.

If you can’t find something or have a specific need like using a non supported version of a particular software, please reach out to the support.