Project Highlight: Diamond MVC Framework

Posted on

Anyone who has been around the D community for longer than an eye blink will have heard of vibe.d, undoubtedly the most widely-used web application framework written in the D programming language. Those same people could be excused if they haven’t yet heard of Diamond, announcements for which have only started showing up in the D forums relatively recently.

According to its website, Diamond is “a full-stack, cross-platform MVC/Template Framework” that’s “inspired by ASP.NET and uses vibe.d for its backend”. Jacob Jensen, the project’s author, explains.

I have always been interested in web development, so one of the few projects I started writing in D was a web server using Phobos’s std.socket. It wasn’t a notable project or anything, but more an experiment. Then I discovered vibe.d and toyed around with it. Unfortunately, coming from a background working with ASP.NET using Razor, I wasn’t a big fan of the Diet templates. So initially I started Diamond as an alternative template engine. However, I ended up just adding more and more to the project and it soon became more and more independent. At this point, you don’t really write a lot of vibe.d code when using it, because the most general vibe.d features have wrappers in Diamond to better interact with the rest of the project.

Development on Diamond began in early 2016, but he put it aside a few months later. Then in October of 2017, after picking up some contract web app work, Diamond was resurrected in its 2.0 form.

I decided I wanted to use D, so I simply did a complete revamp of the project and plan to keep maintaining it.

His biggest hurdle has been keeping the design of the framework user-friendly and minimizing complex interactions between controllers, views, and models.

It was a big challenge to ensure that everything worked together in a way that made it feel natural to work with. On top of that I had to make sure that Diamond worked under multiple build types. At first Diamond was written solely with the web in mind and thus only supported websites and web APIs, but I saw the potential to use the template engine for more than just the web, like email templates. Thus I introduced yet a third way of building Diamond applications, which had to be completely separated from the web part of Diamond without introducing complexity into user code or the build process. By introducing stand-alone support, Diamond is now able to be used with existing projects that aren’t already using it for the web, e.g. someone could use Diamond to extend their existing web pages without having to switch the whole project to Diamond, or simply use Diamond for only a small portion of it.

Aside from the challenge of maintaining user-friendliness, Jacob says he’s encountered only a few issues in developing the framework, most of which came when he was refactoring for the 2.0 release. One in particular is interesting, as his solution for it was a D language feature you don’t often hear much about.

When I was introducing attributes to controllers to avoid manual mapping of actions, it took a while to figure out the best approach to it without having to pass additional information about the controller to its base class.

To demonstrate, Controller subclasses could originally be declared like so:

class MyController(TView) : Controller!TView
{
    ...
}

After he initially added attributes, the refactoring required the base class template to know the derived type at compile time in order to reflect on its attributes. His initial solution required that subclasses specify their own types as an additional template parameter to the Controller template in the class declaration.

class MyController(TView) : Controller!(TView, MyController!TView)
{
    ...
}

He didn’t like it, but it was the only way he could see to make the base class aware of the derived type. Then he discovered D’s template this parameters.

Template this parameters allow any templated member function to know at compile time the static, i.e. declared, type of the instance on which the function is being called.

module base;
class Base 
{
    void printType(this T)() 
    {
        import std.stdio;
        writeln(typeid(T));
    }
}

class Derived : Base {}

void main()
{
    Derived d1 = new Derived;
    auto d2 = new Derived;
    Base b1 = new Derived;

    d1.printType();
    d2.printType();
    b1.printType();
}

And this prints (in modulename.TypeName format):

base.Derived
base.Derived
base.Base

In Diamond, this is used in the Controller constructor in order to parse the UDAs (User Defined Attributes) attached to the derived type at compile time:

class Controller(TView) : BaseController
{
    ...
    this(this TController)(TView view)
    {
        ...

        static if (hasUDA!(TController, HttpAuthentication))
        {
            ...
        }

        static if (hasUDA!(TController, HttpVersion))
        {
            ...
        }
        ...
    }
}

The caveat, and the price Jacob is willing to pay for the increased convenience to users, is that instances of derived types should never be declared to have the type of the base class. When working with templated types in D, it’s idiomatic to use type inference anyway:

// This won't pick up the MyController attributes, as the declared
// type is that of the base class
Controller!ViewImpl controller1 = new MyController!ViewImpl;

// But this will
MyController!ViewImpl controller3 = new MyController!ViewImpl;

// And so will this -- it's also more idiomatic
auto controller2 = new MyController!ViewImpl;

Overall, Jacob has found the transition from C# to D fairly painless.

Most code I was used to writing, coming from C#, is pretty straight-forward in D. One of the pros of D, however, is its compile-time functionality. I use it heavily in Diamond to parse templates, map routes and controller actions, etc. It’s a really powerful tool in development and probably the most powerful tool in D. I also really like templates in D. They’re implemented in a way that doesn’t make them seem complex, unlike in C++, where templates can often seem obscure and cryptic. D is probably the most natural programming language that I’ve used.

Diamond indirectly supports Mongo and Redis through vibe.d, and has its own MySQL ORM interface that uses the native MySQL library under the hood. He has some plans improve upon the database support, however.

I plan to rewrite the whole MySQL part, since it currently uses some deprecated features – it was based on some old code I had been using. Along with that, I plan on implementing some “generic services” that can be used to create internal services in the project, which will of course wrap database engines such as MySQL, Mongo, Redis, etc., creating a similar API between them all and exposing an easier way to implement sharding.

He also intends to add support for textual data formats other than JSON (such as XML) to make Diamond compatible with SOAP or WCF services, add improved support for components in the view, and provide better integration with JavaScript. He also would like to implement an app server for hosting Diamond applications.

Anyone intending to use D for web work today who, like Jacob, has experience using ASP.NET and Razor should feel right at home using Diamond. For the rest, it’s an alternative to using vibe.d directly that some may find more comfortable. You can find the Diamond source, the current documentation, and the in-development official website (for which Jacob is dog-fooding Diamond) all at GitHub.

DConf 2018: Assemblage in Bavaria

Posted on

It’s official! The D Language Foundation has put out a call for submissions for the next iteration of the annual gathering of D programming language enthusiasts. DConf 2018, hosted by QA Systems, is taking place in Munich from May 2nd to the 5th, 2018.

This time around, there’s a focus on growth and outreach. DConf has always been open to all, but past editions largely targeted those already “in the know”. For DConf 2018, the D Language Foundation is actively reaching out, encouraging anyone with little or no D language experience to stop by and see what all the fuss is about.

In the coming months, the D Blog will feature a series of posts related to DConf 2018. To get us started, Andrei Alexandrescu, Vice President and Treasurer of the D Language Foundation, sat down to answer a few questions about the event.


Q: Thanks for taking time out of your schedule for this, Andrei. The first thing I want to get to is the choice of location. At the end of DConf 2017, there was a lot of speculation about where the next edition would be held. We’ve seen two in Menlo Park, California, one in Orem, Utah, and two in Berlin. What led to the choice of Munich?

A: It has a lot to do with my recent visit there. I had mentioned a while ago to our tireless collaborator Sebastian Wilzbach (who studies at both the Technical University of Munich and Ludwig Maximilian University) about the annual classes I teach in neighboring Stuttgart. He suggested I make two trips in one and give a talk in Munich as well.

Once we committed to a date, I was shocked by the earnestness of everybody involved with organizing. The event filled within an hour of opening, in comparable amounts by existing D programmers (there’s a strong D community in Munich) and by curious programmers coming from other languages. There was even some competition among companies willing to host the event.

We ended up holding it at Brainlab’s new headquarters (check it out, they are a great innovator in medical technology). The event was a triumph! The folks in the audience were that combination of smart, receptive, and inquisitive that makes for an amazing interaction. We started at 6:30 and quite a few of us segued into beers, dinner, and of course more chatting, to finally part around midnight.

At that point I thought, Munich sounds like a perfect place for DConf. Later I spoke to my business partner (Andreas Sczepansky, owner of QA Systems) about the great reception the talk got in Munich. He got intrigued and agreed to work with us on DConf 2018. And here we are.

Q: What can attendees expect to see at DConf 2018?

A: We’re counting on a strong technical program, as has been the case in the past events. Also, last year’s day-long hackathon (a largely unstructured “let’s work on cool stuff in small groups” day) was surprisingly successful and enjoyed by everyone involved. So we’re making it bigger and hopefully better this year. It will be on the last day of the event, May 5th.

This year we also want to promote a growth theme. We’re working on bringing a strong outside keynote speaker, and QA Systems will help us to market to companies and grass-roots coders who are currently using other languages. We believe D offers many strategic advantages to the high-tech milieu in Bavaria and beyond.

Q: What do you mean by that? What makes Bavaria special?

A: I noticed there’s a strong IT industry in the area built around automotive, industrial machinery, healthcare, scientific computing, and more. Really serious software with difficult demands and high stakes. We’re talking about systems ranging from memory-constrained embedded systems to high-performance desktop software to large systems that take a long time to design, build, and test. D is all about building fast software, fast. So we have a great opportunity to make the strong case that the D language could help these application domains.

Q: You and Walter Bright have traditionally given the opening and closing keynotes at every DConf. What are you guys planning to talk about this time?

A: I know Walter is considering giving a talk on Project Detente – a multifaceted approach to smooth interoperation with C and C++ that also allows easy incremental migration of large projects from those languages to D. As for me, I haven’t decided yet. I’m really excited by the opportunities opened by this Design by Introspection thing I discussed in my DConf 2017 keynote [Also, see the blog post he wrote about his presentation at Google’s Tel Aviv campus – Ed.].

Q: Last question: what’s the elevator pitch for DConf? If you only had 30 seconds to sell a prospective attendee on the event, what would you say?

A: D is a language with depth. Richness. It has unique solutions to some difficult problems, such as reconciling compile-time computation, partial evaluation, domain-specific languages, and metaprogramming all together in a wholesome manner. Such matters are so fundamental to the way we design, build, and execute our programs that we either consider them solved or unsolvable. Chances are, attending DConf will make you like the D language more. But more importantly, your view of your own métier will be improved regardless of your languages of choice.


Be sure to keep an eye on this space for more details about DConf 2018 as they are released. And if you’re planning to submit a talk, don’t procrastinate. The submission deadline is Feb 25th.

See you in Munich!

DMD 2.077.0 Released

Posted on

The D Language Foundation is happy to announce DMD 2.077.0. This latest release of the reference compiler for the D programming language is available from the dlang.org Downloads page. Among the usual slate of bug and regression fixes, this release brings a couple of particulary beneficial enhancements that will have an immediate impact on some existing projects.

Cutting symbol bloat

Thanks to Rainer Schütze, the compiler now produces significantly smaller mangled names in situations where they had begun to get out of control, particularly in the case of IFTI (Implicit Function Template Instantiation) where Voldemort types are involved. That may call for a bit of a detour here.

The types that shall not be named

Voldemort types are perhaps one of D’s more interesting features. They look like this:

auto getHeWhoShallNotBeNamed() 
{
    struct NoName 
    {
        void castSpell() 
        {
            import std.stdio : writeln;
            writeln("Crucio!");
        }           
    }
    return NoName();
}

void main() 
{
    auto voldemort = getHeWhoShallNotBeNamed();
    voldemort.castSpell();
}

Here we have an auto function, a function for which the return type is inferred, returning an instance of a type declared inside the function. It’s possible to access public members on the instance even though its type can never be named outside of the function where it was declared. Coupled with type inference in variable declarations, it’s possible to store the returned instance and reuse it. This serves as an extra level of encapsulation where it’s desired.

In D, for any given API, as far as the world outside of a module is concerned, module private is the lowest level of encapsulation.

module foobar;

private struct Foo
{
    int x;
}

struct Bar 
{
    private int y;
    int z;
}

Here, the type Foo is module private. Bar is shown here for completeness, as those new to D are often surprised to learn that private members of an aggregate type are also module private (D’s equivalent of the C++ friend relationship). There is no keyword that indicates a lower level of encapsulation.

Sometimes you just may not want Foo to be visible to the entire module. While it’s true that anyone making a breaking change to Foo’s interface also has access to the parts of the module that break (which is the rationale behind module-private members), there are times when you may not want the entire module to have access to Foo at all. Voldemort types fill that role of hiding details not just from the world, but from the rest of the module.

The evil side of Voldemort types

One unforeseen consequence of Voldemort types that was first reported in mid–2016 was that, when used in templated functions, they caused a serious explosion in the size of the mangled function names (in some cases up to 1 MB!), making for some massive object files. There was a good bit of forum discussion on how to trim them down, with a number of ideas tossed around. Ultimately, Rainer Schütze took it on. His persistence has resulted in shorter mangled names all around, but the wins are particularly impressive when it comes to IFTI and Voldemort types. (Rainer is also the maintainer of Visual D, the D programming language plugin for Visual Studio)

D’s name-mangling scheme is detailed in the ABI documentation. The description of the new enhancement is in the section titled ‘Back references’.

Improved vectorization

D has long supported array operations such as element-wise addtion, multiplication, etc. For example:

int[] arr1 = [0, 1, 2];
int[] arr2 = [3, 4, 5];
int[3] arr3 = arr1[] + arr2[];
assert(arr3 == [3, 5, 7]);

In some cases, such operations could be vectorized. The reason it was some cases and not all cases is because dedicated assembly routines were used to achieve the vectorization and they weren’t implemented for every case.

With 2.077.0, that’s no longer true. Vectorization is now templated so that all array operations benefit. Any codebase out there using array operations that were not previously vectorized can expect a sizable performance increase for those operations thanks to the increased throughput (though whether an application benefits overall is of course context-dependent). How the benefit is received depends on the compiler being used. From the changelog:

For GDC/LDC the implementation relies on auto-vectorization, for DMD the implementation performs the vectorization itself. Support for vector operations with DMD is determined statically (-mcpu=native, -mcpu=avx2) to avoid binary bloat and the small test overhead. DMD enables SSE2 for 64-bit targets by default.

Note that the changelog initially showed -march instead of -mcpu in the quoted lines, and the updated version had not yet been posted when this announcement was published.

DMD’s implementation is implemented in terms of core.simd, which is also part of DRuntime’s public API.

The changelog also notes that there’s a potential for division performed on float arrays in existing code to see a performance decrease in exchange for an increase in precision.

The implementation no longer weakens floating point divisions (e.g. ary[] / scalar) to multiplication (ary[] * (1.0 / scalar)) as that may reduce precision. To preserve the higher performance of float multiplication when loss of precision is acceptable, use either -ffast-math with GDC/LDC or manually rewrite your code to multiply by (1.0 / scalar) for DMD.

Other assorted treats

Just the other day, someone asked in the forums if DMD supports reproducible builds. As of 2.077.0, the answer is affirmative. DMD now ensures that compilation is deterministic such that given the same source code and the same compiler version, the binaries produced will be identical. If this is important to you, be sure not to use any of the non-determistic lexer tokens (__DATE__, __TIME__, and __TIMESTAMP__) in your code.

DMD’s -betterC command line option gets some more love in this release. When it’s enabled, DRuntime is not available. Library authors can now use the predefined version D_BetterC to determine when that is the case so that, where it’s feasible, they can more conveniently support applications with and without the runtime. Also, the option’s behavior is now documented, so it’s no longer necessary to go to the forums or parse through search results to figure out what is and isn’t actually supported in BetterC mode.

The entire changelog is, as always, available at dlang.org.

DCompute: Running D on the GPU

Posted on

Nicholas Wilson is a student at Murdoch University, studying for his BEng (Hons)/BSc in Industrial Computer Systems (Hons) and Instrumentation & Control/ Molecular Biology & Genetics and Biomedical Science. He just finished his thesis on low-cost defect detection of solar cells by electroluminescence imaging, which gives him time to work on DCompute and write about it for the D Blog.He plays the piano, ice skates, and has spent 7 years putting D to use on number bashing, automation, and anything else that he could make a computer do for him.


DCompute is a framework and compiler extension to support writing native kernels for OpenCL and CUDA in D to utilize GPUs and other accelerators for computationally intensive code. Its compute API drivers automate the interactions between user code and the tedious and error prone APIs with the goal of enabling the rapid development of high performance D libraries and applications.

Introduction

This is the second article on DCompute. In the previous article, we looked at the development of DCompute and some trivial examples. While we were able to successfully build kernels, there was no way to run them short of using them with an existing framework or doing everything yourself. This is no longer the case. As of v0.1.0, DCompute now comes with native wrappers for both OpenCL and CUDA, enabling kernel dispatch as easily as CUDA.

In order to run a kernel we need to pass it off to the appropriate compute API, either CUDA or OpenCL. While these APIs both try to achieve similar things they are different enough that to squeeze that last bit of performance out of them you need to treat each API separately. But there is sufficient overlap that we can make the interface reasonably consistent between the two. The C bindings to these APIs, however, are very low level and trying to use them is very tedious and extremely prone to error (yay void*).
In addition to the tedium and error proneness, you have to redundantly specify a lot of information, which further compounds the problem. Fortunately this is D and we can remove a lot of the redundancy through introspection and code generation.

The drivers wrap the C API, providing a clean and consistent interface that’s easy to use. While the documentation is a little sparse at the moment, the source code is for the most part straightforward (if you’re familiar with the C APIs, looking where a function is used is a good place to start). There is the occasional piece of magic to achieve a sane API.

Taming the beasts

OpenCL’s clGet*Info functions are the way to access properties of the class hidden behind the void*. A typical call looks like

enum CL_FOO_REFERENCE_COUNT = 0x1234;
cl_foo* foo = ...; 
cl_int refCount;
clGetFooInfo(foo, CL_FOO_REFERENCE_COUNT, refCount.sizeof, &refCount,null);

And that’s not even one for which you have to call, to figure out how much memory you need to allocate, then call again with the allocated buffer (and $DEITY help you if you want to get a cl_program’s binaries).

Using D, I have been able to turn that into this:

struct Foo
{
    void* raw;
    static struct Info
    {
        @(0x1234) int referenceCount;
        ...
    }
    mixin(generateGetInfo!(Info, clGetFooInfo));
}

Foo foo  = ...;
int refCount = foo.referenceCount;

All the magic is in generateGetInfo to generate a property for each member in Foo.Info, enabling much better scalability and bonus documentation.

CUDA also has properties exposed in a similar manner, however they are not essential (unlike OpenCL) for getting things done so their development has been deferred.

Launching a kernel is a large point of pain when dealing with the C API of both OpenCL and (only marginally less horrible) CUDA, due to the complete lack of type safety and having to use the & operator into a void* far too much. In DCompute this incantation simply becomes

Event e = q.enqueue!(saxpy)([N])(b_res, alpha, b_x, b_y, N);

for OpenCL (1D with N work items), and

q.enqueue!(saxpy)([N, 1, 1], [1 ,1 ,1])(b_res, alpha, b_x, b_y, N);

for CUDA (equivalent to saxpy<<<N,1,0,q>>>(b_res,alpha,b_x,b_y, N);)

Where q is a queue, N is the length of buffers (b_res, b_x & b_y) and saxpy (single-precision a x plus y) is the kernel in this example. A full example may be found here, along with the magic that drives the OpenCL and CUDA enqueue functions.

The future of DCompute

While DCompute is functional, there is still much to do. The drivers still need some polish and user testing, and I need to set up continuous integration. A driver that unifies the different compute APIs is also in the works so that we can be even more cross-platform than the industry cross-platform standard.

Being able to convert SPIR-V into SPIR would enable targeting cl_khr_spir-capable 1.x and 2.0 CL implementations, dramatically increasing the number of devices that can run D kernel code (there’s nothing stopping you using the OpenCL driver for other kernels though).

On the compiler side of things, supporting OpenCL image and CUDA texture & surface operations in LDC would increase the applicability of the kernels that could be written.
I currently maintain a forward-ported fork of Khronos’s SPIR-V LLVM to generate SPIR-V from LLVM IR. I plan to use IWOCL to coordinate efforts to merge it into the LLVM trunk, and in doing so, remove the need for some of the hacks in place to deal with the oddities of the SPIR-V backend.

Using DCompute in your projects

If you want to use DCompute, you’ll need a recent LDC built against LLVM with the NVPTX (for CUDA) and/or SPIRV (for OpenCL 2.1+) targets enabled and should add "dcompute": "~>0.1.0" to your dub.json. LDC 1.4+ releases have NVPTX enabled. If you want to target OpenCL, you’ll need to build LDC yourself against my fork of LLVM.

DMD, Windows, and C

Posted on

The ability to interface with C was baked into D from the beginning. Most of the time, it’s something that requires little thought – as long as the declarations on the D side match what exists on the C side, things will usually just work. However, there are a few corner-case gotchas that arise from the simple fact that D, though compatible, is not C.

An upcoming series of posts here on the D blog will delve into some of these dark corners and shine a light on the traps lying in wait. In these posts, readers will be asked to follow along by compiling and executing the examples themselves so they may more thoroughly understand the issues discussed. This means that, in addition to a D compiler, readers will need access to a C compiler.

That raises a potential snafu. On the systems that the DMD frontend groups under the version(Posix) umbrella, it’s a reliable assumption that a C compiler is easily available (if a D compiler is installed and functioning properly, the C compiler will already be installed). The concept of a system compiler is a long established tradition on those systems. On Windows… not so much.

So before diving into a series about C and D, a bit of a primer is called for. That’s where this post comes in. The primary goal is to help ensure a C environment is installed and working on Windows. It’s also useful to understand why things are different on that platform than on the others. Before we get to the why, we’ll dig into the how.

First, assume we have the following two source files in the same directory.

cfoo.c

#include <stdio.h>

void say_hello(void) 
{
    puts("Hello!");
}

dfoo.d

extern(C) void say_hello();

void main() 
{
    say_hello();
}

Now let’s see how to get the two working together.

DMD and C

The DMD packages for Windows ship with everything the compiler needs: a linker and other tools, plus a handful of critical system libraries. So on the one hand, Windows is the only platform where DMD has no external dependencies out of the box. On the other hand, it’s the only platform where a working DMD installation does not imply a C compiler is also installed. And these days, the out-of-the-box experience often isn’t the one you want.

On all the other platforms, the C compiler option is the system compiler, which in practice means GCC or Clang. The system linker to which DMD sends its generated object files might be ld, lld, ld.gold, or any ld-compatible linker. On Windows, there are currently two compiler choices, and neither can be assumed to be installed by default: the Digital Mars C and C++ compiler, dmc, or the Microsoft compiler, cl. We’ll look at each in turn.

DMD and DMC

The linker (Optlink) and other tools that ship with DMD are also part of the DMC distribution. DMD uses these tools by default (or when the -m32 switch is passed on the command line). To link any C objects or static libraries, they should be in the OMF format. Compilers that can generate OMF are a rare breed these days, and while something like Open Watcom may work, using DMC will guarantee 100% compatibility.

The DMC package (version 8.57 as I write) can be downloaded from digitalmars.com. It’s a 3 MB zip file that can be unzipped anywhere. Personally, since I only ever use it in conjunction with DMD, I keep it in C:\D so that the dm directory is a sibling of the dmd2 directory. Once it’s unzipped, it can be added to the path if desired. Be aware that some of the tools DMD and DMC ship with may conflict with tools in other packages if they are on the global path.

For example, both come with Digital Mars make and Optlink, which is named link.exe. The former might conflict with Cygwin, or a MinGW distribution that’s independent of MSYS2 (if mingw32-make has been renamed), and the latter with Microsoft’s linker (which generally shouldn’t be on the global path anyway). Some may prefer just to keep it all off the global path. In that case, it’s simple to configure a command prompt shortcut that sets the PATH when it launches. For example, create a batch file, that looks like this:

echo Welcome to your Digital Mars environment.
@set PATH=C:\D\dmd2\windows\bin;C:\D\dm\bin;%PATH%

Save it as C:\D\dmenv.bat. Right click an empty spot on the desktop and, from the popup menu, select New->Shortcut. In the location field, enter the following:

C:\System\Win32\cmd.exe /k C:\d\dmenv.bat

Now you have a shortcut that, when double clicked, will launch a command prompt that has both dmd and dmc on the path.

Once installed, documentation on the command-line switches for the tools is available at the Digital Mars site. The most relevant are the docs for DMC, Optlink, and Librarian (lib.exe). The latter two will come in handy even when doing pure D development with vanilla DMD, as those are the tools needed to when manually manipulating its object file output.

That’s all there is to it. As long as both dmc.exe and dmd.exe are on the path in any given command prompt, both compilers will find the tools they need via the default settings in their configuration files. For knocking together quick tests with both C and D on Windows, it’s a quick thing to launch a command prompt, compile & link, and execute:

dmc -c cfoo.c
dmd dfoo.d cfoo.obj
dfoo

Easy peasy. Now let’s look at the other option.

DMD and Microsoft’s CL

Getting DMD to work with the Microsoft toolchain requires installing the Microsoft build tools and the Windows SDK. The easiest way to get everything is to use one of the Community editions of Visual Studio. The installer will download and install all the tools and the SDK. The latest is always available from https://www/visualstudio.com. With VS 2017, the installer has been overhauled such that it’s possible to minimize the size of the install more than was possible with past editions. An alternative is to install the Microsoft Build Tools and the Windows SDK separately. However, this is still a large install that isn’t much of a win in light of the new VS 2017 installer options (for those on Windows 8.1 or 10).

Once the tooling is installed, DMD’s configuration file needs to be modified to point its environment variables to the proper locations. Rather than repeat all of that here, I’ll direct you to the DMD installation page at the D Wiki. One of the reasons to prefer the DMD installer over the zip archive is that it will detect any installation of Visual Studio or the Microsoft Build Tools and automatically modify the configuration as needed. This is more convenient than needing to remember to update the configuration every time a new version of DMD is installed. It also offers to install VS 2013 Community if the tooling isn’t found, can install Visual D (the D plugin for Visual Studio), and will add DMD to the system path if you want it to.

It’s a bit of an annoyance to launch Visual Studio for simple tests between C and D. Since it’s not recommended to put the MS tools on the system path, each VS and Microsoft Build Tools installation ships with a number of batch files that will set the path for you (like the one we created for DMC above). The installer sets up shortcuts in the Windows Start menu. There are several different options to choose from. To launch a 64-bit environment with VS 2017 (or the 2017 build tools), find Visual Studio 2017 in the Start menu and select x64 Native Tools Command Prompt for VS 2017. For VS 2015 (or the 2015 build tools), go to Visual Studio 2015 and click on VS 2015 x64 Native Build Tools Command Prompt. Similar options exist for 32-bit (where x86 replaces x64) and cross compiling.

From the VS-enabled 64-bit environment, we can run the following commands to compile our two files.

cl /c cfoo.c
dmd -m64 dfoo.d cfoo.obj
dfoo

In a 32-bit VS environment, replace -m64 with -m32mscoff.

The consequences of history

When a new programming language is born these days, it’s not uncommon for its tooling to be built on top of an existing toolchain rather than completely from scratch. Whether we’re talking about languages like Kotlin built on the JRE, or those like Rust using LLVM, reusing existing tools saves time and allows the developers to focus their precious man-hours on the language itself and any language-specific tooling they require.

When Walter Bright first started putting D together in 1999, that trend had not yet come around. However, he already had an existing toolchain in the form of the Digital Mars C and C++ compiler tools. So it was a no-brainer to make use of his existing tools and compiler backend and just focus on making a new frontend for DMD. There were four major side-effects of this decision, all of which had varying consequences in D’s future development.

First, the DMC tools were Windows-only, so the early versions of DMD would be as well. Second, the linker, Optlink, only supports the OMF format. That meant that DMD’s output would be incompatible with the more common COFF output of most modern C and C++ compilers on Windows. Third, the DMC tools do not support 64-bit, so DMD would be restricted to 32-bit output. Finally, Symantec had the legal rights to the existing backend, which meant their license would apply to DMD. While the frontend was open source, the backend license required one to get permission from Walter to distribute DMD (on a side note, this prevented DMD from being included in official Linux package repositories once Linux support was added, but Symantec granted permission to relicense the backend earlier this year and it is now freely distributable under the Boost license).

DMD 0.00 was released in December of 2001. The 0.63 release brought Linux support in May of 2003. Walter could have based the Linux version on the GCC backend, but as a business owner, and through a caution born from past experience, he was concerned about any legal issues that could arise from his working with GPL code on one platform and maintaining a proprietary backend on another. Instead, he modified the DMD backend to generate ELF objects and hand them off to the GCC tools. This decision to enhance the backend became the approach for all new formats going forward. He did the same when adding support for Mac OS X: he modified the backend to work with the Mach-O object format.

Along with the new formats, the compiler gained the ability to generate 64-bit binaries everywhere except Windows. In order to interface with C on Windows, it was usually necessary to convert COFF object files and static libraries to OMF, to use a tool like coffimplib to generate DLL import libraries in the OMF format, or to create dynamic bindings and load DLLs manually via LoadLibrary and GetProcAddress. Then Remedy Games decided to use D.

Quantum Break was the first AAA game title to ship with D as part of its development process. Remedy used it for their gameplay code, creating their own open source tool to bind with their C++ game engine. Before they could get that far, however, they needed 64-bit support in DMD on Windows. That was the motivator to get it implemented. It took a while (apparently, there are some undocumented quirks in Microsoft’s variant of COFF, a.k.a PECOFF, a.k.a. MS-COFF), but Walter eventually got it done, and support for 32-bit COFF along with it. Again, as he had on other platforms, he modified the backend to generate object files in the new format.

This is why it’s necessary to have the Microsoft toolchain installed in order to produce 64-bit binaries with DMD on Windows. Microsoft’s cl is as close to a system compiler as one is going to get on Windows. There is, however, an option that has not yet been fully explored. It’s a toolchain that can be freely distributed, packaged with a reasonable download size, supports 32-bit and 64-bit output, and is mostly compatible with PECOFF. There is a possibility that it may be investigated as an option for future DMD releases to be built upon.

Going from here

Now that this primer is out of the way, the short series on C is just about ready to go. It will kick off with a brief summary of existing material, showing how easy it is to get D and C to work together in the general case. That will be followed up by two posts on arrays and strings. This is where most of the gotchas come into play, and anyone using D and C in the same program should understand what they are and how to avoid them.