Blog

Continuing the series of Splunk Cluster setup articles with instructions on  Setup of Ubuntu Linux in Azure for Splunk Cluster setup.

Contents:

• Setup VMWare Virtual Machine (Azure services)
• Install and Setup Ubuntu Linux (Azure services)
• Install and Setup Splunk
• Indexes Configuration
• Enable Clustering
• Add second Splunk instance to the cluster
• * Potential problems
• Add third Splunk instance to the cluster (Search Head)

Once Ubuntu Linux VM is provisioned in Azure, in addition to RDP setup we need to attach disks for our data.

I had to have a mix of Windows and Linux Indexers for transition and there is a nuance in having that, more on it later.

Sample setup will include a disk for Hot/Warm, a disk for Cold, and disk for Frozen buckets. [Understanding Buckets][How the indexer stores indexes]

Most of this info came thanks to Azure article. Special to How To Geek for details on fdisk.

Content:

• Attach Disk to VM in Azure Portal
• Initialize Disk and create partition in Linux
• Setup RAID 0 configuration for Hot/Warm buckets

Details:

Attach Disk in Azure portal

1. Select you Virtual machine and navigate to Dashboard tab
2. Click on Attach button and select Attach Empty Disk
3. You can give this disk a distinct name and select the disk size (max 1023GB)
4. Click the check mark to confirm

Initialize Disk and create Partition in Linux

We can do this step via RDP connection we’ve setup in previous step, or we can SSH into our VM.

Method using RDP is a bit tricky as you will have to RDP as root user in order to use Disk Utility (any suggestion how to overcome this limitation, please let me know via Twitter) but we will have to change disk permissions later.

For now let’s move with SSH method.

1. SSH into your server
2. Run command  to get the list of attached disks.

   sudo grep SCSI /var/log/syslog

3. Last one is the one we have attached in the previous step. Note the identifier i.e. [sdc]
4. Now let’s partition this drive with one primary partition [fdisk documentation]

   sudo fdisk /dev/sdc

5. Use n command for new partition
6. Use p parameter for primary partition
7. Accept all default settings for partition number, first cylinder, last cylinder
8. Review settings using p command
9. And accept/write them using w command
10. Create file system on this new partition, make sure you use proper Device Boot value from steps above

    sudo mkfs -t ext4 /dev/sdc1

We now are ready to set up mount point for this partition.

 

We will not mount this drive until Splunk is installed, you will see why.

Setup RAID 0 disk for Hot/Warm buckets

Without jumping too much ahead, I want to review the design architecture of disks I chose for Splunk Setup. Your configuration might vary. Refer to Splunk documentation for more details [Understanding Buckets][How the indexer stores indexes]

• Hot/Warm buckets are searchable buckets and require faster disk
• Cold buckets can reside on slower drive to save money, while Frozen buckets are essentially ‘deleted’ buckets and depending on setup might not even be required.

My setup has 2 virtual drives RAIDed 0 for Hot/Warm buckets, and single drive disks for Cold and Frozen. We’ve setup single drive disk in previous step for Cold and will skip Frozen setup as it is identical to Cold.

1. Attach 2 new empty disks in Azure Portal as outlined above.
2. Install mdadm tool [wiki]

   apt-get install mdadm

3. Run command  to get the list of attached disks.

   sudo grep SCSI /var/log/syslog

4. Note last two disks’ identifiers, we will need it for RAID setup command
5. For RAID 0 setup, issue command

   sudo mdadm --create --verbose /dev/md0 --level=stripe --raid-devices=2 /dev/sde /dev/sdd

6. Now let’s partition just like have done before. Note that our disk identifier is now md0

   sudo fdisk /dev/md0

7. Use n command for new partition
8. Use p parameter for primary partition
9. Accept all default settings for partition number, first cylinder, last cylinder
10. Review settings using p command
11. And accept/write them using w command
12. Create file system on this new partition, make sure you use proper Device Boot value from steps above

    sudo mkfs -t ext4 /dev/md0p1

And we are done! Yo can repeat these steps for any additional disks/drives you would like to use for next step ofSetting up Splunk