DDS Security the Hard(ware) Way - SGX: Part 2 (Micro + Security + SCONE)

**Please note that with the April 2024 release of Connext Professional 7.3 LTS, the functionalities formerly comprising Connext Secure are now available as an optional component to Connext Professional, named Security Extensions. 

This is Part 2 in a six-blog series on this topic. If you missed the Part 1 overview, please read it here.

Development for Software Guard Extensions (SGX) was originally designed by Intel® to be a refactoring process. Each application would be designed from the ground up or rearchitected to partition secrets and secret-managing code from other code, then compiled with the SGX software development kit (SDK) to protect the sensitive code partition. This aligned with Intel’s goal of reducing the trusted computing base (TCB) to the smallest area possible.

With existing applications, however, this is quite a process. In testing, Intel and the United States Airforce Academy refactored an open source PDF viewer in an early access study. The refactoring included adding access control to any part of a document (digital redaction) and an open source Video Teleconferencing (VTC) application (wire to screen encryption). The process took two man years to complete both projects.

As refactoring/recompiling is both a heavy lift and, worse, not always possible due to code access, a number of other projects have come forward to create SGX protected containers/LibOSes and cross compilers. Graphene and SCONE are good examples of projects that are publically available to accomplish these tasks. I have looked at both Graphene and SCONE, and both have pros/cons for end results of a successful build. However, SCONE can already cross compile both RTI Connext® DDS Micro and RTI Connext DDS Security plugins with only tweaks to build scripts and/or create trivial musl library changes.

In addition, SCONE produces a statically-linked application that is executable on a vanilla Ubuntu 16.04 with the SGX driver (and capable CPU) installed.

Graphene requires implementation of at least one new system call (getifaddrs) or changes to DDS to avoid the call, as well as several changes to other DDS calls often done in Connext DDS Micro with more limited operating systems (i.e., nanosleep). Graphene also must be executed as a docker container. As a result, this first focus is on implementing a secure Connext DDS Micro application with SCONE. We will talk more of Graphene later in this blog series.

One of the SCONE projects consists of a cross-compiler that outputs an executable binary that wraps the application in an SGX-protected environment. This cross-compiler statically links against musl rather than GLibC. As a result, we will compile statically all of the components necessary to build our application, including OpenSSL and Connext DDS Micro.

To follow along, you will need RTI DDS Connext Micro 3.0 (including the buildable source), openssl 1.0.2r source, and SCONE. RTI Connext DDS products are available (with licensed access) by contacting RTI; OpelSSL is available at https://www.openssl.org/source/; and SCONE is accessed via SCONE curated images on dockerhub. These SCONE containers are private, and you must obtain access by contacting SCONE via info@scontain.com.

My host system is SGX capable and running Ubuntu 16.04 LTS. You can follow along without an SGX system. If you have an SGX system – and want to use SGX – you must install the Intel SGX driver from https://github.com/intel/linux-sgx-driver. If you are reading this blog, it is assumed that you know how to use docker, docker hub and Linux (or are willing to learn).

To begin, I put both the RTI DDS Connext Micro 3.0 and OpenSSL 1.0.2r in my home directory. I unpacked them in the home directory and end up with two directories:

openssl-1.0.2r
rti_connext_dds_micro-3.0.0

All of the following commands are specific to these directories. Login to docker and ensure you are in your home directory. Run the following commands to start the container in interactive mode. If you are following along without SGX, omit the --device=/dev/isgx from the command.

cd ~
docker run -it --device=/dev/isgx -v "$PWD":/home
sconecuratedimages/crosscompilers:ubuntu

I have found that the container is a bit light on some necessary (and some convenient) tools. To remedy this, install the tools directly.

apt update
apt install -y make default-jre cmake nano less
apt install -y perl --reinstall

Next, let’s compile OpenSSL. The previous reinstall of perl took care of some lacking modules in the config script. To make, test and install OpenSSL to your container, run the following commands. Be forewarned that the test will take some time to complete. For a ballpark benchmark, it took 45 minutes to run the following commands on an i5 NUC.

cd /home/openssl-1.0.2r
./config
make
make test
make install
export OPENSSLHOME=/usr/local/ssl

ln -s /usr/local/ssl/lib/libcrypto.a /usr/local/ssl/lib/libcryptoz.a
ln -s /usr/local/ssl/lib/libssl.a /usr/local/ssl/lib/libsslz.a

The soft links at the end of the command will help with the Makefile in the example we will use. Yes, it is a bit hacky (what do you expect from the security guy?), but it works.

Next, let’s compile RTI DDS Micro 3.0. Run the following commands.

cp /opt/scone/cross-compiler/x86_64-linux-musl/lib/libpthread.a /opt/scone/cross-compiler/x86_64-linux-musl/lib/libnsl.a

PATH=$PATH:/opt/scone/cross-compiler/libexec/gcc/x86_64-linux-musl/7.3.0/
export RTIMEHOME=/home/rti_connext_dds_micro-3.0.0
export RTIMEARCH=sgxLinux_x64gcc

cd /home/rti_connext_dds_micro-3.0.0/

./resource/scripts/rtime-make -DRTI_NO_SHARED_LIB:bool=true -DOPENSSLHOME=/usr/local/ssl --delete --target self --name $RTIMEARCH -G "Unix Makefiles" --build --config Release

At this point, we should have libraries from both OpenSSL and RTI DDS Micro (including security) linked against musl rather than GLibC.

Let’s move on to a test application. Run the following commands.

cd /home/
cd rti_connext_dds_micro-3.0.0/example/unix/C/
cd HelloWorld_dpde_secure
rm -rf objs
make

If all has gone well, we will have both a publisher and a subscriber in the following directory: /home/rti_connext_dds_micro-3.0.0/example/unix/C/HelloWorld_dpde_secure/objs/sgxLinux_x64gcc/. Let’s run it to see what we have.

SCONE_VERSION=1 SCONE_HEAP=87108864 ./objs/sgxLinux_x64gcc/HelloWorld_publisher

The output should look something like:

export SCONE_QUEUES=1
export SCONE_SLOTS=256
export SCONE_SIGPIPE=0
export SCONE_MMAP32BIT=0
export SCONE_SSPINS=100
export SCONE_SSLEEP=4000
export SCONE_KERNEL=0
export SCONE_HEAP=87108864
export SCONE_STACK=81920
export SCONE_CONFIG=/home/jason/sgx-musl.conf
export SCONE_ESPINS=10000
export SCONE_MODE=hw
export SCONE_SGXBOUNDS=no
export SCONE_VARYS=no
export SCONE_ALLOW_DLOPEN=no
export SCONE_MPROTECT=no
Revision: 4be39d5943d5c15e11fa17055b859de4a25c0288 (Thu Aug 23 14:14:04 2018 +0200)
Branch: cf-java-fix (dirty)
Configure options: --enable-shared --enable-debug --prefix=/home/christof/GIT/subtree-scone/built/cross-compiler/x86_64-linux-musl

Enclave hash: 14fa1810e1d35799ba9910243cab89660b7146f96babb97a32caef9c06b3c9a2

[1555446711.154091000]ERROR: ModuleID=0 Errcode=17 X=1 E=0 T=1
osapi/posixThread.c:96/OSAPI_Thread_get_policy: sysrc=38
# Identity CA, : file:security/ca/ca.pem
# Permissions CA, : file:security/ca/ca.pem
# PEER certificate : file:security/ca/certs/publisher.pem
# PEER key : file:security/ca/certs/publisher_key.pem
# XML governance : file:security/xml/governance.p7s
# XML permissions : file:security/xml/permissions_publisher.p7s

[1555446711.159431000]ERROR: ModuleID=0 Errcode=17 X=0 E=1 T=1
<not found>/<not found>:-1/<not found>:

[1555446711.159704000]ERROR: ModuleID=0 Errcode=17 X=0 E=1 T=1
<not found>/<not found>:-1/<not found>:

[1555446711.197874000]ERROR: ModuleID=0 Errcode=17 X=0 E=1 T=1
<not found>/<not found>:-1/<not found>:
Hello World! (0)
Hello World! (1)
Hello World! (2)
Hello World! (3)
Hello World! (4)

Don’t be afraid of the reported errors. They are informational only and are due to the Connext DDS Micro libraries attempting to check for the ability to manipulate thread priory. Musl does not support this. The reported errors can be removed by editing posixThread.c at the line indicated (in this case, line 96) to simply set the default policy without doing the check and, thus, eliminating the reporting of the error (again, hacky, I know).

So this is all well and good, but what have we accomplished? We can run our DDS application inside an SGX enclave, inside of the SCONE container. While this is useful, it’s even more useful that our new application binary is self contained and can be run in a vanilla Ubuntu 16.04 environment.

Let’s try it. Since we created the executable in a mapped directory, it will persist when we exit the container. We need to set up a config file for our SCONE environment. Create it with the following command.

tee ~/sgx-musl.conf << EOF
Q 4
e -1 0 0
s -1 0 0
e -1 1 0
s -1 1 0
e -1 2 0
s -1 2 0
e -1 3 0
s -1 3 0
EOF

Now we can run our application:

cd ~/rti_connext_dds_micro-3.0.0/example/unix/C/HelloWorld_dpde_secure/

SCONE_CONFIG=~/sgx-musl.conf SCONE_VERSION=1 SCONE_HEAP=87108864 ./objs/sgxLinux_x64gcc/HelloWorld_publisher

Check to see if they communicate:

SCONE_CONFIG=~/sgx-musl.conf SCONE_VERSION=1 SCONE_HEAP=87108864 ./objs/sgxLinux_x64gcc/HelloWorld_subscriber

We have now compiled our first SGX-protected application.

This is certainly just a “hello world” type example and does not include any information about how to safely store information on disk, nor in memory, nor remotely attest applications. However, we now have a basis to work from.

In Part 3, I will examine Hardened DDS Services in SGX environments.

About the author

Dr. Jason Upchurch is a Cyber Security Research Scientist at RTI. Jason came to RTI from Intel Corporation where he was a Platform Security Architect and the Principal Investigator for a joint security laboratory with the US Department of Homeland Security, the United States Air Force Academy, and Intel. In addition to his work at RTI and Intel, Jason's security background includes serving as the most senior technical representative of the Cyber Protection and Digital Forensics Group at General Dynamics AIS and the Chief/General Manager of the Intrusions Group at the Defense Cyber Crime Center (DC3). Jason's operational service highlights include leading the forensic efforts in the early advanced persistent threat (APT) intrusions as well as several multi-million dollar financial intrusion investigations. Jason's research projects have included software similarity, compiled code reuse detection, hardware root of trust security, and hardware assisted attack detection. He received his Ph.D. in Engineering - Security at the University of Colorado, Colorado Springs.