NSBogan

TeamCity Kotlin REST Client

2019-06-24T00:00:00+00:00

Whichever CI server you use for development, being able to access it programmatically via REST API is an essential capability for implementing all kinds of automation. In this article you will learn how to utilize TeamCity REST API using TeamCity Kotlin REST Client.

Let’s start by setting up some goals.

I want to be able to access my TeamCity server using Kotlin programming language.

The goal is set and to help me with the task JetBrains provides a REST client which I’ve mentioned earlier. So all I have to do is write a simple Kotlin script that uses REST client and implement whichever automation I want, right? Well, except that I’m not a Java or Android developer and building Kotlin source code is totally new to me, so this article will cover some basics.

⚙️ Setup

I’m working on a Mac OS X machine, so I’ll start by getting Gradle Build Tool installed.

Next, I’ll create a new directory for my project as well as the main Kotlin file and a Gradle wrapper.

From now on I should be running all Gradle commands using the wrapper, e.g. ./gradlew build. Wrapper will be saved in gradle/wrapper folder. This folder can be committed to source control which removes the dependency on global Gradle installation and makes the project self-contained.

I want to use Kotlin to write build scripts, so I create build.gradle.kts file in the project root with the following contents:

This is to make sure I’m using exact version of Gradle and it doesn’t autoupdate.

🔨 Implement

This may sound unusual, but in this case I choose to write the Kotlin code first and then make it compile and run. Based on the example on REST client project page, I came up with the following:

In a nutshell this code connects to TeamCity using username and password provided as input arguments, then fetches latest build information for a specified build configuration and prints it out to console. Check inline comments for more details.

👷‍♂️ Build

After a lot of trial an error I came up with the the following build file:

Check the inline comments for detailed information. This build script builds Kotlin application targeted for running on top of JVM. The application depends on TeamCity REST client.

Note that you have to install Java version 1.8 to build and run the application.

It’s time to build it:

🏃‍♀️ Run

I can now run the Kotlin application:

After customizing input arguments I am able to see something like this in the logs:

That’s just the beginning! I can now use nice modern programming language to automate various CI/CD tasks on TeamCity.

For a full example project see this repository.

Xcode Build Settings in Depth

2019-05-12T00:00:00+00:00

Getting to the rock bottom of Xcode build settings.

If you have ever done any Mac OS or iOS development, you eventually had to deal with Xcode build settings.

So what are those are what do we know about them?

For a standard iOS project there’s roughly 500 build settings grouped into around 50 categories. These build settings control virtually every single aspect of how you app is built and packaged. At the very least, build settings is what makes Debug build so different from Release build.

In this article I’ll mostly focus on build settings that control the behavior of Apple Clang and Swift compilers and the linker.

🗄 Background Check

Build settings are not as simple as they may seem.

For starters, there are project and target level build settings as well as default OS settings. The default OS build settings are inherited on a project level and project level build settings are inherited on a target level.

Then there are .xcconfig files, which can be used to represent build settings in plain text format. The xcconfigs can be set on target and project level and add two more levels to inheritance flow.

To make things even more complicated, you can include other xcconfigs using C-like #include statements.

And then there’s much more to build settings and xcconfigs.

To get familiar with all of the above, I’d highly recommend to start with The Unofficial Guide to xcconfig files. Check out the rest of this amazing blog for even more hands-on information about understanding and managing build settings.

⬇️ Level Down

Now let’s get one more level down. When it comes to compiling and linking the source code, Xcode build system manages 3 main tools under the hood:

Clang Cxx compiler for compiling C/C++ and Objective-C/C++ code
Swift compiler
Linker to link all object files together

Each of those tools has its own set of command line flags. Clang compiler and linker flags are documented here. swiftc --help command provides list of Swift command line flags. Surprisingly, I failed to find online documentation similar to Clang.

When a build setting is set in Xcode UI, Xcode then translates it to appropriate flags for underlying tools. For example, setting GCC_TREAT_WARNINGS_AS_ERRORS to YES will add -Werror flag for Clang Cxx compiler.

Similarly, setting SWIFT_TREAT_WARNINGS_AS_ERRORS to YES will add -warnings-as-errors flag to all invocations of Swift compiler.

Finally, some build settings are translated into flags for all three tools, for example, enabling code coverage using CLANG_ENABLE_CODE_COVERAGE = YES build setting, will translate into the following^*:

-fprofile-instr-generate and -fcoverage-mapping for Clang compiler
-profile-coverage-mapping and -profile-generate for Swift compiler
-fprofile-instr-generate for linker

^* Technically, it takes more than just setting CLANG_ENABLE_CODE_COVERAGE to enable code coverage, more details to come.

So, how does Xcode know which flags to map build settings to? Where is this mapping information stored and can it be extracted?

⚙️ Xcode Specs

The answer to the question in previous section is Xcode Specs or xcspecs.

Xcspecs are ASCII plist files with .xcspec file extension stored deep inside Xcode.app bundle. Xcode uses these specs to render build settings UI and to translate build settings to command line flags.

There are xcspecs for Clang compiler (Clang LLVM 1.0.xcspec), Swift compiler (Swift.xcspec) and linker (Ld.xcspec) as well as a number of xcspecs for core build system and other tools. These xcspecs reference each other and work together as a system.

Each xcspec contains specification for one or more tools, for example, Swift xcspec contains specification for Swift compiler tool, while Clang LLVM xcspec contais specifications for Clang compiler, analyzer, couple of migrators and an AST builder tool.

A tool specification includes such details as name, description, identifier, executable path, supported file types and more.

🔘 Options

In context of this article we are most interested in Options array entry of the tool’s specification, which is where information about build settings is stored. For example, the value of SWIFT_EXEC is stored as an option and is used by Xcode to resolve ExecPath = "$(SWIFT_EXEC)"; statement:

All build settings (options) have Name and Type properties. Name defines the build setting name, e.g. SWIFT_OPTIMIZATION_LEVEL.

Build settings may also have a Category, Description and DisplayName properties. For example, display name for SWIFT_OPTIMIZATION_LEVEL is Optimization Level.

A few build settings are hidden and never appear in Xcode UI. Some of them have corresponding comment in the code like // Hidden., but not all.

Types

There are 6 different types of build settings:

String
StringList
Path
PathList
Enumeration
Boolean

ℹ️ String and Path

A build setting of String type has, well, a string value.

Note that string value doesn’t have to be quoted using double quotes. As a matter of fact, some build setting that have integer values are represented using String type with default value set to 0, but not to "0".

Path type is identical to String when used in xcspecs.

My guess is that Path type build settings are handled differently with regards to escaping whitespaces and other special characters. They also get special treatment when resolving wildcard characters like *

ℹ️ StringList and PathList

StringList and PathList are used to represent list of string and path values correspondingly.

If not provided, default value is an empty list.

You can tell a list type in Xcode UI because it allows multiple values input:

ℹ️ Enumeration

Values of Enumeration type have a fixed list of values (cases), defined using Values key, for example:

Another good example is build settings that have YES_AGGRESSIVE or YES_ERROR value on top of YES and NO. These build settings are declared as enumerations too:

ℹ️ Boolean

Finally, values of Boolean type have either YES or NO value.

Note that only YES and NO are correct values for Boolean type, but not "YES" or "NO".

🗺 Command Line Flags Mapping

As we already know, a lot of build setting values map to different command line flags. The mapping information is defined in xcspecs using one of the following keys:

CommandLineArgs
CommandLineFlag
CommandLinePrefixFlag
AdditionalLinkerArgs

ℹ️ Command Line Arguments

CommandLineArgs key-value entry is used to map build setting value to a list of command line arguments.

Scalar Types

A good example is SWIFT_MODULE_NAME build setting of String type:

The $(value) is resolved to current build setting value, e.g. MyModule value is mapped to -module-name "MyModule" Swift compiler flag.

List Types

For list types each build setting value in the list is mapped to one or more command line flags, for example:

If the value of CLANG_ANALYZER_OTHER_CHECKERS is "checker1" "checker2", it will be mapped to the following:

Enumeration Type

For enumerations types, mapping is defined for each enumeration case, for example:

ansi value is mapped to -ansi compiler flag.
compiler-default maps to no flags.
All other enumeration values map to -std=$(value) compiler flag. Note the use of "<<otherwise>>", this is similar to default: enum switch in C-like languages.

Boolean Type

Boolean types are mapped just like enumerations with 2 YES and NO cases.

ℹ️ Command Line Flag

CommandLineFlag is used to prepend command line flag to the value of build setting.

For example, SDKROOT is defined like so:

So if the value of SDKROOT is iphoneos, the corresponding Clang compiler flag will be -isysroot iphoneos.

In a way CommandLineFlag is a shorthand for using CommandLineArgs. I.e. in SDKROOT example, CommandLineFlag = "-isysroot"; could be replaced with CommandLineArgs = ("-isysroot", "$(value)");.

Handling of list, enumeration and Boolean types is similar to handling CommandLineArgs too. For example, given the definition:

The value like SYSTEM_FRAMEWORK_SEARCH_PATHS = "A" "B" "C" will be mapped to:

Enumerations deserve a special mention, because build flag mapping can be defined next to the value. A good example is MACH_O_TYPE:

ℹ️ Command Line Prefix Flag

CommandLinePrefixFlag maps build setting value to itself prefixed with a build flag.

It works just like CommandLineFlag, the only difference is that there’s no space between the build flag and the build settings value.

A good examples are LIBRARY_SEARCH_PATHS and FRAMEWORK_SEARCH_PATHS build settings of list type, where each list entry is mapped to -L"$(value)" or -F"$(value)" correspondingly.

Another similar build setting often used by developers is OTHER_LDFLAGS.

Finally, whenever you see a flag like -fmessage-length=0 it’s most likely mapped using prefix flag rule.

ℹ️ Additional Linker Flags

Certain Swift or Clang compiler build settings map not only to compiler flags, but to linker flags as well. Those build settings have an additional AdditionalLinkerArgs key-value pair, for example:

In this example, the -fobjc-arc flag will be added both to Clang compiler and linker invocations.

The way mapping works for different build setting types is identical to handling of CommandLineArgs.

References

As I’ve mentioned earlier, build settings from different xcspecs can reference each other.

Some build settings have a DefaultValue property. It can reference other build settings, e.g. DefaultValue = "$(BITCODE_GENERATION_MODE)";. Default value of some build settings is defined by referencing other build setting:

The references can be nested as well, for example:

Here $(DEPLOYMENT_TARGET_SETTING_NAME) will be first resolved into a value like SOME_SETTING and then $(SOME_SETTING) is resolved once again to a final value. Similar to how build settings are resolved in xcconfigs.

Build settings reference can be used inside CommandLineArgs and all other mapping key-value entries.

Conditions

Conditions are defined using Condition key-value pair and are used to control when certain build setting is enabled or not.

For example, SWIFT_BITCODE_GENERATION_MODE build setting only makes sense when ENABLE_BITCODE is set to YES:

Conditions have a C-like syntax, although string values do not have to be quoted. Such boolean operators as !, &&, ||, == and != can be used:

Other Properties

There are other properties, such as

Architectures - defines which target architectures the build setting is applicable for.
AppearsAfter - controls the order in which 2 specific build settings appear in Xcode UI.
FileTypes - list of applicable file types.
ConditionFlavors - must have something to do with the way conditions are checked.
Some other flags I didn’t have time to fully investigate yet.

👩‍🏫 Example

Let’s have a look at one build setting example and see how we can apply all the knowledge from this article to figure out which flags it will map to.

The build setting is CLANG_ENABLE_CODE_COVERAGE and it enables one very important feature - code coverage.

CLANG_ENABLE_CODE_COVERAGE is defined in Clang LLVM 1.0.xcspec but strangely maps to no flags at all…

However, CLANG_ENABLE_CODE_COVERAGE is referenced by CLANG_COVERAGE_MAPPING.

So now we know the Clang compiler flags added to compiler invocation when code coverage is enabled.

Additionally, there’s another build setting that controls the linker flag:

It also comes with a very detailed comment.

Finally, CLANG_COVERAGE_MAPPING is also defined in Swift.xcspec:

Now all the pieces of the puzzle come together and it’s clear how Xcode does the mapping.

The only problem is that both CLANG_COVERAGE_MAPPING and CLANG_COVERAGE_MAPPING_LINKER_ARGS are hidden and don’t show up in Xcode UI. So how do those get set if they don’t reference CLANG_ENABLE_CODE_COVERAGE in their default value?

The code coverage can be enabled by editing the scheme in Xcode UI or by passing -enableCodeCoverage YES to xcodebuild invocation

Xcode will then set both CLANG_COVERAGE_MAPPING and CLANG_COVERAGE_MAPPING_LINKER_ARGS to YES under the hood.

👷‍ Application

OK then, so it’s more or less clear how build settings are resolved, but what’s the practical application?

Well, there’s the purely academic application where you get to know how things work, which occasionally will come handy when you have hard time figuring out some build settings mess.

While the official Xcode Build Settings reference page is a good resource to use, the Extended Xcode Build Settings reference like this one includes information about compiler and linker flags, build settings cross-reference and more.

There are other ways to use this knowledge.

Let’s say you want to try out alternative build system like Buck or Bazel. Those build systems are gaining popularity these days. Buck was created by Facebook while Bazel came from Google. Other companies such as Uber, Airbnb and Dropbox use Buck; while Lyft is using Bazel to build their mobile apps.

Let’s further assume that over the years you have created and maintained a number of amazing xcconfig files. Those xcconfigs have all the compiler and linker build settings fine-tuned for your use. While moving to tools like Buck, you can’t just bring xcconfigs over, instead you’d want to translate xcconfigs to compiler and linker flags and then use those with Buck.

Well, now you have all the knowledge to do so. You’d need to start with reading and resolving xcconfigs and then map resolved build settings to build flags. It’s not a straightforward task and may take a while to implement. Luckily for you, there’s a Fastlane plugin that does just that.

Like many unofficial tools, this plugin is reverse engineering the ways Xcode works, so use it at your own risk.

Fastlane for Enterprise

2019-04-07T00:00:00+00:00

Learn how to use Fastlane in enterprise environment.

In case you were wondering, no - Fastlane doesn’t have an Enterprise offering. It always was and I hope will be an open-source product.

What the article really is about is how to use Fastlane in enterprise environment. Specifically those Fastlane actions that need connection to external URLs.

Enterprise Environment

If you are one of the lucky devs, who just “connect to Internet directly”™️, you may as well just skip this article and get back to enjoying Getting Things Done.

If, my unlucky friend, the words like company proxy and company firewall sound familiar, then you may find the rest of the article useful.

So, in very simple terms for non-DevOps engineers like me, working in a big enterprise company usually means that there’s at least one proxy server sitting between you, the developer, and outside world, aka The Internet. Proxy will make sure to authenticate you securely, just to know who you are; and it will do other important things, like peeking inside every single zip archive you download. Why? Because it can…

To make sure you don’t visit harmful websites like GitHub Gists (🤯), there’s also a network firewall thrown into the mix. The firewall will only allow connections to the whitelisted endpoints.

Whitelist

If you are the one tasked with a job of “Uploading ipa file to TestFlight from CI server”, the first thing to do is to request access to Apple endpoints via your company proxy and firewall. Mind you, the journey you are about to embark on will make Homer’s Odyssey look like a pleasant weekend trip. Depending on the complexity of your company’s IT infrastructure and level of support, you’d better collect all the information on your side first and check it twice, before submitting any requests.

Now, Spaceship is the core part of Fastlane that enables access both to Apple Developer Center and to App Store Connect. The list of API endpoints used by Spaceship is a good starting point for the request you are about to submit.

I’d even recommend to go one step further and request to whitelist access to all URLs on apple.com domain. It will make your setup more future proof, in case Apple adds new APIs or changes existing private APIs, which they do quite often.

The final list of domains to whitelist should be like this:

*.apple.com/* - for all kind of Apple stuff.
*.devimages.apple.com.edgekey.net/* - this is not strictly required for communicating to dev portal or App Store Connect, but this is where Fastlane can download iOS Simulator images.
*.mzstatic.com/* - so that Fastlane can download screenshots of your app from App Store Connect.

The port to request access for is 443 - for all HTTPS traffic.

🤖 Service Account

Before you go ahead with submitting new request for your IT infrastructure change, make sure you have a service account as well.

Unlike normal user accounts, service accounts are not tied to a particular employee. This is very important. When a person leaves a company, their account gets eventually wiped and CI pipelines get broken, so the lesson is: never use personal accounts for CI setup. Service accounts don’t ask for a pay rise, don’t chuck a sickie, and they just don’t quit.

Another good thing about service accounts, they normally don’t have the password expiry policy set. You don’t have to update your CI configuration with a new password every 90 days or so.

If you don’t have a service account, request one then. Yes, it may mean you have to open yet another request to some other DevOps team, but who said enterprise was easy?

Using Fastlane behind the Proxy

Fast-forward N weeks, where N is a random large non-negative number, and you have the following at your disposal:

A service account username and password, and maybe even an email, if you are super lucky.
Let’s assume the username is part of Active Directory Domain and comes in a au\username form, where au is the domain.
Let’s also assume the password is just password.
All requested domains are whitelisted on all levels of intranet.
You service account can authenticate to the company proxy using username and password and access whitelisted domains.
Let’s assume the proxy host name is company-proxy and it listens on port 8080 for all incoming connections.

Curl Test

First off, run a simple curl test to make sure everything works:

The --proxy-user and --proxy options are self-explanatory, though note the backslash escaping as \\.

By saying --proxy-anyauth we tell curl to work with any authentication scheme that the proxy supports.

Most likely, all outgoing network traffic on your dev machine is signed with a company’s own self-signed Root Certificate Authority (CA) certificate, so we use --insecure option to convince curl to accept this self-signed certificate.

The self-signed root CA and authentication scheme will play very important role further on.

The output of the command should be something like this:

Unauthenticated

Request ID: ABCABCBACBACXYZXYZXYZZ37.0.0

Which is good. It means the request went all the way to appstoreconnect.apple.com.

🔏 Root CA and SSL Certificates

Just like with curl, self-signed root CA certificate will not be accepted by Fastlane either. While running Fastlane, you won’t have any option like the curl’s --insecure flag. Instead you need to add your internal root CA certificate to rubygems path, because Fastlane is a Ruby gem:

You may have to use sudo if you are using system Ruby version, though I’d recommend to use RVM or rbenv to manage your rubies.

If you are using RVM you need to run the following as well:

Now you can use this Ruby script to test SSL connection.

E.g. just run it directly like this:

💲 Environment Variables

Next thing you will notice is that Fastlane doesn’t have any option for passing proxy information. Luckily, Fastlane is being a good citizen and respects all the /http[s]_proxy/i environment variables, namely:

HTTP_PROXY
HTTPS_PROXY
http_proxy
https_proxy

Remember this “Gang of Four”, nothing else will cause you as much pain as this bunch!

Somewhere in your ~/.bash_profile or other appropriate shell profile, set these environment variables, for example:

Note that proxy URL is set in the following format:

http://<domain>\<username>:<password>@host:port

Without username and password, you wouldn’t be able to authenticate with your proxy.

Now give it a try, run the simple download_screenshotsdeliver action:

Answer all the prompts and see if it worked. I bet it did not!

Long story short, Fastlane is using faraday HTTP client Ruby library, which uses Kernel.URI method, which can’t handle backslash \ in username.

The solution is to use percent-encoding for \, i.e. %5c:

You should be able to download screenshots now, though that’s just a first tiny step towards the final goal. Well, actually, you can do a lot of things now, such as registering devices, managing App IDs and provisioning profiles, and more. But I’d assume you also want to upload a new ipa to TestFlight, so meet The Transporter then…

🚎 Transporter

Transporter aka iTMSTransporter is a Java-based command-line tool used to “upload things to App Store”, including app metadata and new ipa builds.

Installing Transporter

So how do you get a copy?

Well… according to Apple, you first login to your developer account and then click the download link like this one.

But wait! You can’t download it unless you have Admin or Technical role.

I mean, “Why, Apple?! Why?!” 😱

Transporter is available as part of Xcode installation anyways, why make things so hard?

Why not make it available at Apple Developers Downloads at the very least?

Anyways, if you choose to use copy bundled with Xcode, then you can find it at

/Applications/Xcode.app/Contents/Applications/Application Loader.app/Contents/itms

Just grab the whole directory.

If you installed it via Mac OS package, then get it at

/usr/local/itms

Patching Transporter

If you wonder: “Why do I need a copy of Transporter? Can’t Fastlane just use it as is?”

Good question, you don’t need a copy of Transporter, you can use the one available inside Xcode or in your /usr/local, but you would have to patch it before using anyways.

Given that Transporter is a Java-based command-line app, it has the same issue as curl or any Ruby gem would have with your company’s internal self-signed root CA certificate - it won’t trust it. Apple didn’t bother to add a command line or configuration option of any kind to Transporter to trust self-signed certificates.

Thankfully, Transporter isn’t just a Java-based app, it comes with it’s own version of Java runtime bundled up inside itms/java. Try running itms/java/bin/java -version to see more info.

Now, the itms/java folder also contains lib/security/cacerts file, which is a keystore with CA certificates. So you need to add your company certificate to cacerts and itms/java/bin/keytool is just the right tool to do that.

Here ${ROOT_CA_PATH} is path to the certificate file, which you can locate directly or find it in OS X keychain

Now you’ve convinced Transporter to trust your root CA, but you are not there yet, because you also need to teach Transporter some respect towards your new and shiny proxy, which is where the trouble awaits for you… again.

Transporter and Proxy

Well, now that we know that Transporter is a Java-based app, we look closer inside itms folder and find itms/java/lib/net.properties - a Java properties files used to configure JVM that Transporter runs in.

The group of proxy variables is exactly what we need:

You don’t even have to modify the net.properties file, Fastlane reads the value of special DELIVER_ITMSTRANSPORTER_ADDITIONAL_UPLOAD_PARAMETERS environment variable and uses it to configure Transporter, for example in your bash script add the following export statement:

But will it work? By now you’ve been there enough times to know the answer - “No”. 😜

Proxy configuration needs to include username and password for Basic authentication.

You can try to add au%5cusername:password@company-proxy as the host value, but it will not work.

Removing jdk.http.auth.tunneling.disabledSchemes=Basic limitation from net.properties won’t help either.

While trying to google you may find mention of these properties:

Those are part of Apache HTTP Client JVM options though and are not part of default Oracle’s network properties, also Transporter doesn’t respect them.

🚧 Feels like another roadblock.

Another Proxy

The solution this time is to create yet another proxy (as if there wasn’t enough problems with them already!).

This “man in the middle” local proxy will take care of passing authentication information to your company proxy. The credentials no longer have to be hard-coded in proxy URL. This way the command line tools like Transporter can use local proxy for network connections and local proxy doesn’t require any type of authentication.

There is a number of proxy tools available for Mac OS. In this example I’ll be using CNTLM.

I’ll briefly mention that I also tried TinyProxy and ProxyChains-NG. TinyProxy just wouldn’t let me use \ or %5c in configuration file, while with ProxyChains-NG I just eventually gave up after a number of failed attempts to make it work.

So, CNTLM, a fast proxy for NTLM authentication implemented in C.

Easy to install.

Easy to configure, just put the following in cntlm.conf:

Now run this command to generate password hashes:

Type in your account password, copy hashes and append them to cntlm.conf:

Then run it:

Finally, configure proxy environment variables:

Give it a go!

What do you mean by “It doesn’t work!”?. Something about proxy returning invalid challenge?

Well, that just means you company proxy doesn’t support NTLM authentication and only supports Basic authentication. That needs to be changed by requesting IT support again, I’m afraid.

Now the curl test should finally work and your configuration looks like this:

Upload `ipa`

You are all set now to run one of the most common and recurring tasks in Continuous Delivery process for an iOS app - upload a new build to TestFlight.

Start by setting up the basic lane using pilot action.

To skip the password prompt, set FASTLANE_PASSWORD environment variable. It could be a secret parameter in your CI server configuration or you could save it in OS X keychain as AppleID.Password and read like so:

It’s important not to confuse Apple ID used as username with Apple ID of the app. Latter is the unique identifier the app is assigned in App Store, that’s why it is often referred to as Adam or Adam/Apple/App ID. You can find out Adam ID of the app by navigating to it in App Store Connect and grabbing the last part of URL.

Then again, app_identifier parameter is actually the Bundle Identifier stored under CFBundleIdentifier in the application’s Info.plist.

In this example, we don’t want to submit new build for review yet and we don’t want to wait for processing either, thus both skip_submission and skip_waiting_for_build_processing are set to true.

Note, that with most of Fastlane actions, each action parameter is shadowed by an environment variable. For example, instead of setting ipa parameter you can set PILOT_IPA environment variable. To see full list of parameters for an action, run bundle exec fastlane action <action_name>, for example:

Try to run the lane now, after setting up all the environment variables, of course:

Fastlane Session

If you plan to use similar action on CI server, then Two-factor authentication (2FA) enforced by Apple will become an issue. To avoid 2FA on CI servers, you need to set the FASTLANE_SESSION environment variable as described here.

Whenever using 2FA the browser will prompt you if you want to trust this particular browser. Doing so will generate a long-living session token (30 days) which will then be picked by Fastlane’s spaceauth command. This way you will only have to update your CI server configuration once every 30 days. There may be some ways to keep the session token up to date by running scheduled CI jobs that perform Spaceship login, though I haven’t tested that approach yet.

Summary

By the end of this journey you are able to upload new iOS app builds to TestFlight from the enterprise network, e.g. from a CI build agent.

It’s amazing how such a basic task requires so much extra effort when dealing with all the overhead inherent to most large companies.

I hope it was worth the effort though. Once it all set up and runs smoothly, there are so many other apsects of iOS development that can be automated using tools like Fastlane.

Xcode Build Phases and Environment

2019-04-04T00:00:00+00:00

How to teach Xcode to respect the Environment.

If you’ve ever done iOS development you’ve surely used Xcode Build Phases. One of the tasks that a build phase can perform is running a shell script, and that’s where you can face one peculiar problem…

The Problem

The problem is that shell scripts ran in Xcode build phases do not source the user shell profile.

Try to put the following in your shell profiles~/.profile:

Then run the build phase by building Xcode project.

None of the messages from shell profiles will show up.

The reason is that a shell ran from Xcode build phase is non-interactive shell.

It is not actually not a bug, but an expected behavior, which may confuse you a bit though.

The Example

As an example, let’s say you’ve got a Ruby script which you want to run as part of Xcode build phase.

The script happens to be using some 2.5.x Ruby language features, which are not available in 2.3.x.

You are also using RVM to install and use version 2.5.x.

Now add ruby --version to the build phase, build the project and check the output. On Mojave OS X the output will be 2.3.7, which is the system Ruby version.

None of the RVM environment usually defined in shell profile got loaded into script.

One solution to this problem is to use login shell.

It’s as simple as adding -l to the “Shell” parameter of the build phase, for example /bin/sh -l. However, it will not work for Bourne shell (sh).

But the it will work for the Bourne-again shell (bash), so you could change “Shell” parameter to say /bin/bash -l.

You will see this in the logs now:

Loading ~/.bash_profile

That works, but is not ideal. If you have a lot of developers on the team, they may have all kinds of things happening in their ~/.bash_profile which you don’t necessarily want to run in Xcode build phase. All you want is only load RVM environment, nothing else.

Other developers may as well use zsh or some other kind of shell.

Build Phase Scripts

Another way to tackle this problem is to source just the things that matter as part of each build phase script.

For example, to use RVM for loading Ruby version in non-interactive shell, you could use this code:

Here we first add rvm command to the PATH. Then we read Ruby version used by the project from .ruby-version file, which is one of the common setups. Finally we get a shell script containing all the environment variables using rvm "${RUBY_VERSION}" do rvm env --path and source (load) that script into current shell session.

Now you can put this code to a dotenv file called .env.ruby, for example. Each build phase shell script that requires Ruby can now add source .env.ruby line.

While this approach may look like it requires more work, it is still a better one for a couple of reasons:

You only load the things you need to run your script, not all the things from user’s shell profile
The Ruby version is selected based on the project’s .ruby-version file, while with login shell approach the default Ruby version set for RVM globally is loaded and you’d have to run extra code to switch to correct version

Tips

A few tips to using shell script build phases.

Script Errors

Add -e flag for build phase to fail if shell script returns error code other that 0, e.g. /bin/sh -e.

Managing Build Phase Scripts

Use the following convention to manage you build phase scripts:

There must be one shell script per build phase.

For example, put all your build phase scripts in a directory named build-phases, e.g. build-phases/codecheck.sh, build-phases/codegen.sh, etc.

In Xcode build phase UI instead of having an inline script, you have this now:

You can even move the Ruby environment loading line inside the build phase script to keep it down to a one-liner.

It’s a much better approach for a number of reasons:

If you modify the build phase script, it’s much easier to review it in pull request as a change of .sh file, compared to when you have to review it as part of pbxproj UTF-8 ASCII plist file changes with all the newline and other escape sequences added into the mix.
You can reuse build phase scripts outside of Xcode, e.g. if you try to use alternative build system like Buck.

Swift Refactoring - Summary

2019-02-06T00:00:00+00:00

A summary for the series of articles about implementing Swift Local Refactoring action.

If you came here after going through previous 6 articles, you may have a few questions to ask:

Was it worth it?

Sure it was.

I learned the basics of working with Swift code base. How to build it, how to write tests and debug the code.

This whole exercise took me one tiny step closer to understanding the rest of the Swift project.

Would you do it again?

The answer is “Yes” and “No”.

No, I would probably not try to implement another refactoring action via swift-refactor. Instead, I’d look into using libSyntax library.

Yes, if I had more time to spare I’d look at one of the many Swift starter tasks.

Swift Refactoring - Debugging

2019-02-05T00:00:00+00:00

Debugging implementation for Swift Local Refactoring action.

Previous - Implementation Part 2

Ninja

If you have built swift-refactor tool using ninja, then you’d have to use command line lldb debugger to debug the code. So after building swift-refactor, just run lldb in the terminal:

Next, tell the debugger which executable to debug (swift-refactor in this case):

Then set the breakpoint, e.g. for isApplicable implementation of RefactoringActionCollapseNestedIfStatement class:

Now tell lldb to run the executable with -source-filename and -pos arguments

The debugger has now stopped on line 1714 and is ready for your further input:

-> 1714

Same debug commands you’d use in Xcode UI work in command line:

Xcode

If you choose to develop using Xcode, debugging is even easier.

Edit the swift-refactor scheme

and add launch arguments:

Set breakpoint and hit “Run”:

Next - Summary

Swift Refactoring - Implementation - Part 2

2019-02-04T00:00:00+00:00

Implementing Swift Local Refactoring action.

Part 2 of 2.

In the previous part we’ve implemented isApplicable check.

The very same findNestedIfStatements can be reused to implement refactoring transformation.

// 1 The first thing to do is check if refactoring can be applicable. Note that we return true to abort the refactoring transformation.

// 2 Next we create a buffer to write transformed code into and then create an output stream for writing to the buffer.

// 3 A collapsed if statement will still be an if statement, so we write the if keyword to output stream using kw_if keyword token.

// 4 Now we need to write the combined conditions list to output stream. Conveniently, conditions in the condition-list can be joined together using commas.

Grammar of an if statement

if-statement → if condition-list code-block else-clause opt

else-clause → else code-block else if-statement

condition-list → condition condition , condition-list

Loop through conditions collected while finding nested if statements:

Get the range of the condition statement SC in the original source code:

Get the string representation of the condition statement from the original source code:

SM here is an instance of SourceManager available to each refactoring action implementation.

Finally write the condition string into output stream joining it with the comma if needed.

On first iteration separator will be an empty string while 2nd and consequent statements are joined by ", " string.

// 5 Now is the time to write the then statement shared by all collapsed if statements. This is the reason we saved LastThenStatement while finding nested if statements. Again, we get the string representation of the then statement and write it to output stream.

Then statement already contains opening and closing braces ({ and }), so no need to add those.

// 6 By this time the new transformed code is saved to output stream OS. Final step is to replace original code with the new code using Source Manager SM.

First step is to get the range of the untransformed source code.

Here we take range from the Start of the FirstIfStmt to the End of the then statement.

That range is then transformed into source range, e.g. range in Source Manager SM’s coordinates:

Finally, the code in SourceRange is replaced with contents of Buffer and we return false to indicate successful transformation (🤷‍♂️)

Full version of Refactoring.cpp.

The implementation is ready to be tested. If anything goes wrong, then it’s time to debug.

Next - Debugging

Swift Refactoring - Implementation - Part 1

2019-02-04T00:00:00+00:00

Implementing Swift Local Refactoring action.

Part 1 of 2.

Previous article - Tests

Add Refactoring Kind

The chosen “Collapse Nested If Statement” refactoring is a cursor-based refactoring, so the first thing to do is to add this code to RefactoringKinds.def:

This macro will generate bare bones for new RefactoringActionCollapseNestedIfStatement class.

A new -collapse-nested-if-statement command line option needs to be added to swift-refactor tool in swift-refactor.cpp:

Now, to finish off the refactoring action implementation, the following two methods have to be implemented in Refactoring.cpp.

Both methods require current cursor info for implementation. While isApplication takes it as a CursorInfo input argument, for performChange current cursor info is available as a member of auto-generated RefactoringActionCollapseNestedIfStatement class.

Is Applicable

The way I approached this implementation task is asking myself a question first.

When is “Collapse Nested If Statements” refactoring action applicable?

The obvious answer is:

When there are more then 1 nested if statements.

So the approach is to find nested if statements at current position and count them.

Let’s first define a simple structure that will hold information about nested if statements:

It’s very straightforward, we keep track of the first if statement (FirstIfStmt) and last then statement (LastThenStmt).

One important thing about collapsible nested if statements is that they all share a single then statement, they wouldn’t be collapsible otherwise.

We also save all the if statement conditions into Conditions list.

To decide whether a refactoring action is applicable, we need to have non-null first if and last then statements and more than 1 condition in the list:

With this new structure in mind, we can define a new helper method that finds all nested if statements under the cursor and then use it to implement isApplicable:

Find Nested If Statements

This is the part where we finally get down to using cursor info provided to us by refactoring engine.

It’s easier to reason about the code in terms of declarations, statements and conditions. For example, for a code like this:

The high level view could be this:

The (very rough) Abstract Syntax Tree sketch for this code:

If the cursor is pointing at the start of the first if statement, then the AST we need to analyze looks like this:

So the very first thing to check is to make sure cursor info is pointing at the start of the statement (StmtStart), the refactoring can’t be applied otherwise:

The returned empty NestedIfStatementsInfo() indicates that refactoring cannot be applied because NestedIfStatementsInfo().canProceed() is false.

Walk the Tree

Next thing to do is to start walking Abstract Syntax Tree.

// 1 For that we create a new type called NestedIfStatementsFinder, which inherits from SourceEntityWalker - the base class for walking the source code tree.

// 2 We also initialize an instance of NestedIfStatementsFinder called Walker.

// 3 Additionally, our walker should collect nested if statements info into the Result property.

// 4 and // 5 Finally tell walker to walk the AST starting from CursorInfo.TrailingStmt and return the result.

Next let’s have a look at some of the SourceEntityWalker interface.

It has a handful of methods to walk the AST, specifically it can walk the statement, declaration or expression.

For our implementation we have used bool walk(Stmt *S).

The type also offers customization points:

These methods can be overridden in types inheriting from SourceEntityWalker. When true is returned, the walker will keep walking the tree, otherwise it will stop.

For out implementation, we choose to override walkToStmtPre, which is called when walker is about to walk the statement.

Let’s walk this code step by step:

// 1 We are only interested in nested if statements, so when statement kind (S->getKind()) is not StmtKind::If then there’s no point to continue (return false).

// 2 Now that we checked the statement kind, we can safely typecast it to IfStmt.

// 3 We want to keep track of the first if statement in the Result variable, so save it if this is the first if statement we encounter.

// 4 If current if statement has an else statement, it cannot be collapsed, so there’s no point to keep going.

// 5 We also need to keep track of last then statement, which will be useful while applying refactoring transformation.

// 6 If the then statement of current if statement has more than one elements, then the refactoring cannot be applied from this point on and need to stop.

Here’s an example of Swift code:

The if a < 2 if statement has 2 elements in the then statement: print(a) and if b > 1 statements.

// 7 Keep track of the if statement conditions in Conditions list:

// 8 Finally, recursively walk the first statement inside the then-block.

Test

Now it’s a good time to run the tests created in previous article to make sure implementation of isApplicable works as expected.

Next - Implementation - Part 2

Swift Refactoring - Tests

2019-02-03T00:00:00+00:00

Using TDD approach to implement Swift Local Refactoring action.

Previous article - Setup

TDD or Test-driven development is a perfect approach when you face a new and unfamiliar code base.

It surely helped me, I was able to wind up a large number of tests and then iterate on actual implementation to make those tests pass.

However, learning how to write tests for Swift source code is a challenge by itself. The project is using lit or LLVM Integrated Tester and you can find more details in Apple’s own Testing Swift documentation.

Test Refactoring Kind

I’ll be making changes to the swift-refactor tool in Swift toolchain. For now I will treat that tool as a black box, i.e. I have no idea how the refactoring action is implemented.

What I do know though, is that if I give swift-refactor tool a file and specify a cursor position in that file, then it will print out a list of applicable refactoring actions at that cursor position.

So I’ll start by creating a new test file at swift/test/refactoring/RefactoringKind/collapse_nested_if_statement.swift with the following content:

Now, if I point a cursor at row 4 and column 4 in this file, I expect my new refactoring action to be applicable. If I name the refactoring action “Collapse Nested If Statement”, then this text should be the output of the following command:

So my test case is this:

Given that I run swift-refactor for collapse_nested_if_statement.swift file, row 4 and column 4, I expect the output to contain "Collapse Nested If Statement" string.

All I have to do now is write this test case using lit command. This is how it looks:

This code is put directly into collapse_nested_if_statement.swift file and lit will pick it up from there.

Here’s the detailed breakdown of the test spec:

The CHECK-COLLAPSE-NESTED-IF-STATEMENT is also defined in collapse_nested_if_statement.swift like so:

So the refactoring action first begins (Action begins), then the Collapse Nested If Statement output is written to standard output and then the action ends (Action ends).

The special -NEXT syntax is how you can define multi-line strings.

More examples of refactoring kind tests.

Cursor Position Reference

You have probably spotted a small problem with -pos=4:4 argument of the test command.

If the tests are modified, refactored or moved around, then the actual cursor position of the start of test code can change. When there’s dozens of tests in the single file, fixing up all the impacted -pos= arguments can be quite a challenge.

That’s where using cursor position reference is really helpful.

All you have to do is add a block comment like so

Now you can use "2-statements" string as a reference to the cursor position:

If the test2Statements is relocated in file or some changes happen to formatting or indentation, the tests will still point to the correct cursor position and will require no changes.

Test Refactoring Transformation

Next thing to test is the refactoring transformation itself.

The structure of these tests is like this:

Take the test fixture Swift file that contains code to refactor.
Pass it to swift-refactor and capture the output.
Compare output to expected code using diff utility.

I have created a new CollapseNestedIfStatement folder under swift/test/refactoring, then created basic.swift file and Outputs/basic directory.

basic.swift is where the lit tests will live and Outputs/basic is where expected outputs will be created.

Similar to refactoring kind test, I write the following Swift code in basic.swift:

After the refactoring action has been applied, I expect the code to look like this:

so this is what I put into Outputs/basic/2-statements.swift.expected.

Now I have to write lit test spec in basic.swift:

The test begins by cleaning up output directory with rm -rf %t.result && mkdir -p %t.result command. %t in this case is expanded to the test name.

Next the test runs %refactor command which expands to swift-refactor. The command is almost identical to the one in refactoring kind tests, except that this time I pass -collapse-nested-if-statement to tell swift-refactor to apply this particular refactoring action.

The -collapse-nested-if-statement command line option hasn’t been implemented yet, but remember that this is TDD.

The output of swift-refactor is saved to %t.result/2-statements.swift.

Finally, the output is compared to expected Swift code using diff utility.

-u is used to compare 3 lines of unified context, while -B is used to ignore blank lines.

%S is expanded to the parent directory of the current (basic.swift) file.

That’s the basic approach, next step is to add more test code and expected outputs. Here’s an example.

Running Tests

Once you have enough tests and made your first changes to implement the new refactoring action, you can rebuild only the swift-refactor tool:

where SWIFT_BUILD_DIR is set to location of Swift build folder, for example build/Ninja-RelWithDebInfoAssert+swift-DebugAssert/swift-macosx-x86_64/.

To speed up the development process even further, you can use --filter option of lit.py to run selected tests only. For example, to run only refactoring kind tests from collapse_nested_if_statement.swift:

Or to run refactoring transformation tests from CollapseNestedIfStatement/basic.swift:

Now you are all set for the usual “Build, Test, Repeat” development cycle.

Next - Implementation - Part 1

Swift Refactoring - Setup

2019-02-02T00:00:00+00:00

Getting Swift sources and setting up build environment to implement Swift Local Refactoring action.

Previous article - Intro

When it comes to cloning source code and building it, there’s not that much that I can add to the Swift repository README, but I’ll try to highlight some moments that I find to be important.

Latest Xcode

I can’t stress enough how important this step is!

If you use wrong version of Xcode you will be banging your head against the wall trying to figure out why the build is OK, the tests pass, but you changes do not show up in the Xcode when you choose your new toolchain.

Don’t forget to use latest Xcode when building from command line, for example, if you installed latest Xcode 10.2 beta to /Applications/Xcode-10.2.app, then run this in terminal

Stable Source

If you want to use source code that has passed Swift CI then, after pulling in all the code, you can customize your update-checkout command like so

If the build was tagged, it means it passed all the tests and had a successful Xcode toolchain build. It may be a good idea to work on top of a stable code.

Building Xcode Toolchain

Apple docs just tell you to run ./utils/build-toolchain $BUNDLE_PREFIX command to build a new toolchain. It works, however, the command takes really long time and doesn’t seem to take advantage of incremental builds when you re-run it.

Instead, you can package a toolchain using build artifacts created with build-script.

This and this articles provide more details. Both recommend to use the following command to build Swift code:

Once the build is over, you can use this script to create a toolchain in just few seconds.

Set the SWIFT_SOURCE_PATH to root of your Swift sources location and run the script from that location. Optionally customize CONFIGURATION and BUILD_DIR.

Next - Tests

Swift Refactoring - Intro

2019-02-01T00:00:00+00:00

Introduction to a series of articles detailing how to implement a new type of Swift Local Refactoring.

So you thought about contributing to Swift code base but never knew where and how to start? Well then, Swift Local Refactoring is a very good place to start.

In a series of articles I’ll take you through my experience with implementing a single local Swift refactoring action. The refactoring action I chose as an example is “Collapse Nested If Statements”.

In short, it changes code like thisinto thisThe steps:

Setup Swift build environment.
Follow TDD approach and Write tests first.
Implement Part 1 refactoring action.
Implement Part 2 refactoring action.
Debug the code.
Summary and my thoughts on Xcode’s refactoring feature.

P.S.

You may notice that Collapse Nested If Statements refactoring action is already implemented and available in latest versions of Xcode.

A disclaimer though, I did not implement that action, but I surely took learnings from the current implementation.

The major difference between an action currently available in Xcode and the one described in these articles, is that current Xcode’s version collapses only 2 statements at a time, so if you have 3 or more nested statements, you have to repeat the action to collapse them all.

(Reload to play gifs)

While the action implemented in these articles collapses all nested if statements at once.

Next - Setup

Async Swift Scripting

2015-10-08T00:00:00+00:00

A trick to use asynchronous callbacks in Swift scripts.

I was really inspired by this talk by Ayaka Nonaka. I personally believe that writing scripts in Swift will become A Thing very soon. It’s already happening for Mac OS X, the upcoming Linux compiler will bring it to a next level. There’s already plenty of useful frameworks available via CocoaPods or Carthage. The only thing that’s missing is a decent package manager for Swift frameworks, something like Homebrew. Swift Package Manager (SPAM) sounds like a nice name :)

Anyway, I wanted to use Alamofire in a simple Swift script. So I have copy-paste-edited sample code from their GitHub page and saved it as a Example.swift file.

Build Alamofire

To run this script I need to build Alamofire framework first. There are two ways to do that: using CocoaPods or Carthage.

Before I go on, it’s important to specify versions of the tools I use.

Xcode 9.2
cocoapods gem version 1.4.0
cocoapods-rome gem version 0.8.0
carthage version 0.28.0

CocoaPods

Start with a Podfile that looks like this:

Note that a file named .swift-version must exist in your working directory. The contents of the file are “4.0” indicating Swift language version to use.

Next run pod install to build the frameworks.

%{ gist 4654b77f53c1982ceee1de7e0f73b772 %}

You may want to use bundle exe pod install if you installed gems with Gemfile and bundler. Now you have Alamofire.framework ready for use in Rome directory.

Carthage

Start with a Cartfile.

Then build.

Note the --platform mac option. The option is not really well documented, but it’s extremely important in this case. It tells carthage to build only Mac OS X targets, and that’s exactly what you need for Swift scripting.

You should now have Alamofire.framework ready for use in Carthage/Build/Mac directory.

Run

Time to run the script. To point Swift compiler to location of 3rd party frameworks use -F option and make sure you put it before the name of the Swift file.

And the output is…

Done

Wait a sec. How come? Well, that’s because…

It’s Async!

Yes, the callback from Alamofire is asynchronous. So the script finishes execution before it gets the response callback from Alamofire.

That means we have to keep the script alive and kicking until we get all async callbacks. You have probably thought about semaphores or mutexes right now. Good guess, but that won’t work. Consider this pseudo-code.

The problem is that callback block (closure) is dispatched to the same queue it was originally enqueued from. This is the case for Alamofire and I’m pretty sure for most of the libraries with async callbacks.

WAIT(MUTEX) code will lock the main queue and UNLOCK(MUTEX) line will never be executed.

Run Loop

The answer to this particular problem is Run Loop. Each OS X or iOS application has a main run loop that keeps the app alive and reacts to all kinds of input sources, such as timer events or selector calls. As a matter of fact, our Swift script has a run loop too, all we have to do is to keep it running until all async callbacks are received. The draft solution looks like this:

%{ gist 0f51d4036a48ca31f1178f14aab66e65 %}

In this example we get current run loop (runLoop) and then keep it running with help of runMode(_: beforeDate:) method. According to the documentation this method will return YES if the run loop ran and processed an input source or if the specified timeout value was reached; otherwise, NO if the run loop could not be started.

That’s the main difference from using mutexes or semaphores. runMode doesn’t block main queue, it just puts run loop to sleep until specified time in the future (for 0.1s in this example) and while asleep the run loop can be woken up by an input source. Asynchronous call to our JSON response closure is exactly the type of input source that can wake up a sleeping run loop, so each time runMode returns YES we also check for value of keepAlive and if it’s false, that means we have handled our async callback and the script can stop its execution.

Swift Script Runner

To make the task of writing scripts with async callbacks easier, I have created a SwiftScriptRunner framework. Here’s how you’d use it:

Then in Example.swift:

You can call lock() multiple times before wait(), just make sure you balance each lock() with unlock() to avoid deadlocks.

Code Coverage for iOS (Xcode 7)

2015-09-21T00:00:00+00:00

Create code coverage reports for iOS unit tests using new Xcode 7 code coverage feature.

The Old Way

I’ll start with back reference to another post I wrote earlier, which describes the process of getting code coverage reports using good old gcov.

To try out the old approach checkout this repository and run the scripts.

# Run tests with gcov instrumentation.
./test-gcov.sh

# Generate Cobertura coverage report (output is gcov-report.xml).
./gcovr.sh

# Generate HTML report (output is lcov-reports).
./lcov.sh

There’s nothing new inside those scripts, same stuff as described in the previous blog post. Run these with Xcode 6 and make sure that everything works almost as expected.

Don’t worry if you already trashed Xcode 6 and switched to Xcode 7. You still can run all the same scripts. The only problem is that lcov.sh will fail, that’s not critical but is a first sign of trouble.

The real problem is that there’s no coverage information available for Swift code. That’s because there are no gcda and gcno files generated for Swift.

# For Model.swift
Model.d
Model.dia
Model.o
Model.swiftdeps

# For XCTestCase+GCovFlush.m
XCTestCase+GCovFlush.d
XCTestCase+GCovFlush.dia
XCTestCase+GCovFlush.gcda
XCTestCase+GCovFlush.gcno
XCTestCase+GCovFlush.o

So Apple is slowly deprecating gcov after all.

Profile Data

What should we use to get test coverage reports for Swift code?

Apple is switching to a new Coverage Mapping Format and Profile Data format.

LLVM toolset comes with a number of tools to work with profile data format, specifically llvm-cov. This tool can analyze profile data and instrumented app binary and emit code coverage data in more human-friendly format. But first we need to get profile data generated and app binary instrumented.

Gather Profile Data

With Xcode 7 it’s a much easier task. Instead of specifying 3 build settings xcodebuild now has a single flag called -enableCodeCoverage and all you have to do is set it to YES.

xcodebuild test \
    -project MixAndMatchTests.xcodeproj\
    -scheme MixAndMatchTests \
    -sdk iphonesimulator \
    -configuration Debug \
    -enableCodeCoverage YES

The test-profdata.sh from GitHub repository is more detailed version of the script below. Feel free to just run ./test-profdata.sh.

Setting -enableCodeCoverage flag to YES is essentially the same as checking the “Gather coverage data” checkbox in Xcode scheme settings.

Convert Profile Data

OK, so now profile data is generated and we have an instrumented binary available at out disposal. The question is where do we find the binary and profile data?

Both instrumented binary and profile data are sitting inside derived data directory. We can get derived data path from build settings.

BUILD_SETTINGS=build-settings.txt

# Get build settings
xcodebuild -project MixAndMatchTests.xcodeproj \
    -scheme MixAndMatchTests \
    -sdk iphonesimulator \
    -configuration Debug \
    -showBuildSettings > ${BUILD_SETTINGS}

# Project Temp Root ends up with /Build/Intermediates/
PROJECT_TEMP_ROOT=$(grep -m1 PROJECT_TEMP_ROOT ${BUILD_SETTINGS} | cut -d= -f2 | xargs)

Let’s look for profile data first. From Apple forums we know that we are looking for Coverage.profdata file, so let’s just find it.

PROFDATA=$(find ${PROJECT_TEMP_ROOT} -name "Coverage.profdata")

Looking for binary is similar, we use the fact that it’s sitting inside app bundle, e.g. MixAndMatchTests.app/MixAndMatchTests.

BINARY=$(find ${PROJECT_TEMP_ROOT} -path "*MixAndMatchTests.app/MixAndMatchTests")

Note: You shouldn’t use -derivedDataPath or CONFIGURATION_BUILD_DIR option when running xcodebuild for testing. Having build directory and derived data directory in custom locations will cause some problems for open source tools which I’ll talk later in this article.

OK, so we got both path to binary and path to profile data, let’s feed those to llvm-cov now.

xcrun llvm-cov show \
    -instr-profile ${PROFDATA} \
    ${BINARY}

What you see now is a detailed coverage data. To get a short summary, let’s use report option instead of show.

xcrun llvm-cov report \
    -instr-profile ${PROFDATA} \
    ${BINARY}

# Output
Filename                    Regions    Miss   Cover Functions  Executed
-----------------------------------------------------------------------
...usr/include/MacTypes.h         0       0    nan%         0      nan%
...sr/include/objc/objc.h         0       0    nan%         0      nan%
...r/include/sys/_types.h         0       0    nan%         0      nan%
...tchTests/AppDelegate.m        17       6  64.71%         7    42.86%
...DetailViewController.m         8       8   0.00%         4     0.00%
...MasterViewController.m        30      25  16.67%        10    40.00%
...MatchTests/Model.swift         1       0 100.00%         1   100.00%
...MatchTests/ObjCModel.m         1       0 100.00%         1   100.00%
...ixAndMatchTests/main.m         3       0 100.00%         1   100.00%
-----------------------------------------------------------------------
TOTAL                            60      39  35.00%        24    41.67%

Yes! A nice colorized (at least for me) output! Check llvm-cov-show.sh script for a cleaner version of the shell script.

For now please ignore the fact that some system and test files are included in report, we will filter them out later. In the meantime we have achieved our first goal and converted profile data into some kind of test coverage report.

GCov Report

So we have a coverage report, but how useful is it? Well, it’s not much useful as it is.

The main reason for generating coverage report is to be able to feed it to your favorite CI server and enjoy a nicely formatted and browsable version of it. Ever more, configure your build jobs to be marked as stable, unstable or failed if test coverage is below a certain threshold.

As it happens, none of the popular CI servers seem to have plugins for parsing Profile Data yet. So we have to convert it to something that CI servers can digest, such as Cobertura coverage report.

LLVM toolset doesn’t support such conversion, but there is a glimpse of hope.

A brief search lead me to this Stack Overflow page with further reference to Slather ruby gem. Slather is designed to take care of all your testing tasks including generating coverage report. In this article I’ll only use it for converting profile data to Cobertura format.

There is a pull request #92 (still open at the moment of updating this post) with changes to support this new feature. Let’s build this gem from source and see if it’s up to job.

First we need to create a custom Gemfile.

# Gemfile.
source 'https://rubygems.org'
gem 'slather', :git => "https://github.com/viteinfinite/slather.git", :branch => "feature-profdata"

Then install Slather from this branch.

bundle install

And we are good to go.

bundle exec slather coverage \
    --input-format profdata \
    --cobertura-xml \
    --ignore "../**/*/Xcode*" \
    --output-directory slather-report \
    --scheme MixAndMatchTests \
    MixAndMatchTests.xcodeproj

Now check slather-report directory and you got your cobertura.xml file ready to be fed to CI server plugin! There are options other than --cobertura-xml, such as --simple-output, --html and others.

Let’s use simple output option and compare it to Xcode output.

MixAndMatchTests/AppDelegate.m: 16 of 33 lines (48.48%)
MixAndMatchTests/DetailViewController.m: 0 of 23 lines (0.00%)
MixAndMatchTests/MasterViewController.m: 19 of 63 lines (30.16%)
MixAndMatchTests/Model.swift: 3 of 3 lines (100.00%)
MixAndMatchTests/ObjCModel.m: 3 of 3 lines (100.00%)
MixAndMatchTests/main.m: 5 of 5 lines (100.00%)
Test Coverage: 35.38%

Fastlane

If you are a fan of fastlane (I am), then have a look at project Fastfile. It includes a basic example of how you can use scan and slather actions to get coverage reports.

Caveats

I’m adding this section as an update for this post. There’s a number of things you have to do to have proper coverage for Swift code.

Enable Testability

First of all enable testability for the main target. This is controlled by ENABLE_TESTABILITY build setting that has to be set to YES. In fact, enabling this flag for unit tests target causes no trouble. This flag will allow you to import Swift code from main target in unit tests code in this way:

@testable import MixAndMatchTests

Don’t include main target files to Unit Tests target

This is another mandatory step to get coverage for Swift code. Otherwise you will get a lot of warnings like this in the test log:

# For Swift files
objc[7995]: Class _TtC14MixAndMatchTests4Model is implemented in both
<derived-data-path>/MixAndMatchTests.app/MixAndMatchTests and
<derived-data-path>/MixAndMatchTests.xctest/MixAndMatchTests.
One of the two will be used. Which one is undefined.

# For Objective-C files
objc[7995]: Class ObjCModel is implemented in both
<derived-data-path>/MixAndMatchTests.app/MixAndMatchTests and
<derived-data-path>/MixAndMatchTests.xctest/MixAndMatchTests.
One of the two will be used. Which one is undefined.

This is not only annoying, but also will result in useless code coverage reports generated.

Don’t test Swift code using Objective-C code

This is rather a consequence of first two changes. Since Swift files are not part of test target, there is no generated code for these Swift files in the Swift umbrella header, and that means you can’t use this Swift code from Objective-C.

So you have to test Swift with Swift to get coverage reports.

Avoid using legacy and new coverage formats together

If you coverage results look totally off and you get tons of message in test log, that look like this:

ObjectiveC.gcda: cannot merge previous GCDA file: corrupt arc tag (<some hex address>)

Then you are most-likely mixing both test approaches together, which is not recommended. Make sure you have disabled legacy flags GCC_GENERATE_TEST_COVERAGE_FILES and GCC_INSTRUMENT_PROGRAM_FLOW_ARCS, when you want to use Profile Data. Have a look at these discussions: 1, 2, 3.

Thanks to @GUL- for help with this tricky stuff.

Equatable NSObject With Swift 2

2015-06-21T00:00:00+00:00

[Swift 2.0]

Subtle differences in implementing Equatable protocol for NSObject subclasses in Swift 1.2 and 2.0.

The best way to read this post is in Xcode playground, so go ahead and Download Playground

Equatable

Instances of the type that conform to Equatable protocol can be compared for value equality using operators == and !=.

When adopting Equatable, only the == operator is required to be implemented. The standard library provides an implementation for !=.

This was pretty much a copy-paste from documentation. Let’s try it in practice.

Swift Structs

If you just have a simple struct, you will not be able to compare two instances of it.

struct CardStruct {
    let rank: Int

    init(rank: Int) {
        self.rank = rank
    }
}

Uncomment next line to see the following error:

Error: Binary operator == cannot be applied to two CardStruct operands

//print(CardStruct(rank: 4) == CardStruct(rank: 4))

To fix it, make sure that your structure conforms to Equatable protocol first.

struct EquatableCardStruct: Equatable {
    let rank: Int

    init(rank: Int) {
        self.rank = rank
    }
}

Then implement == operator as prescribed by Equatable.

func ==(lhs: EquatableCardStruct, rhs: EquatableCardStruct) -> Bool {
    return lhs.rank == rhs.rank
}

Now two instances of EquatableCardStruct are equal if they have the same rank.

print(EquatableCardStruct(rank: 5) == EquatableCardStruct(rank: 5))
// true

print(EquatableCardStruct(rank: 5) == EquatableCardStruct(rank: 3))
// false

Since collection types are equatable by default, you can equate two arrays of EquatableCardStructs as well with no additional effort.

let equatableCardStructs1 = [EquatableCardStruct(rank: 1), EquatableCardStruct(rank: 2), EquatableCardStruct(rank: 3)]
let equatableCardStructs2 = [EquatableCardStruct(rank: 1), EquatableCardStruct(rank: 2), EquatableCardStruct(rank: 3)]

print(equatableCardStructs1 == equatableCardStructs2)
// true
print([EquatableCardStruct(rank: 2)] == [EquatableCardStruct(rank: 5)])
// false

NSObject Subclass

So it all works well for structs, but what if you are dealing with NSObject subclass? Let’s declare one to start with.

class CardObject: NSObject {
    var rank: Int

    init(rank: Int) {
        self.rank = rank
    }
}

Now pay close attention to next line.

With Swift 1.2 (Xcode 6.3.2) it will work fine, uncomment and see for yourself.

With Swift 2.0 (Xcode 7) it will produce a compiler error.

Error: Redundant conformance of ‘CardObject’ to protocol ‘Equatable’

// This is "line 93" for future reference
//extension CardObject: Equatable {}

Usual implementation for ==.

func ==(left: CardObject, right: CardObject) -> Bool {
    return left.rank == right.rank
}

Now let’s try to equate some objects.

Look at the next two lines while using Swift 1.2. Notice that commenting and uncommenting line 93 doesn’t change the behavior. Despite the fact that CardObject is not declared to conform to Equatable, the code still works. Most likely this is due to the fact that all code is in one playground, this == operator would most definitely fail if it were in another file.

print(CardObject(rank: 5) == CardObject(rank: 5))
// true

print(CardObject(rank: 5) == CardObject(rank: 3))
// false

This is where we can verify previous statement. Again, toggle comments on line 93 to see how result of the next == changes.

let cardObjects1 = [CardObject(rank: 1), CardObject(rank: 2), CardObject(rank: 3)]
let cardObjects2 = [CardObject(rank: 1), CardObject(rank: 2), CardObject(rank: 3)]
print(cardObjects1 == cardObjects2)
// false when line 93 commented
// true otherwise

So when CardObject is conforming to Equatable explicitly equating collections works as expected, otherwise it doesn’t, most likely isEqual: method of NSObject is used, which compares object hashes.

But give it a try with Swift 2.0 now. First of all you will be told that NSObject already conforms to Equatable, so line 93 has redundand conformance. Once you comment that line cardObjects1 and cardObjects2 are not equal any more, because it’s all back to comparing hash values now.

In order to fix this problem, you will have to override isEqual method.

class EquatableCardObject: NSObject {
    var rank: Int

    init(rank: Int) {
        self.rank = rank
    }

    override func isEqual(object: AnyObject?) -> Bool {
        if let rhs = object as? EquatableCardObject {
            return rank == rhs.rank
        }
        return false
    }
}

This class is almost identical to CardObject, except for the isEqual method. Another difference is that there’s no global == implemented to compare 2 instances of EquatableCardObject.

Nevertheless the comparison works.

print(EquatableCardObject(rank: 5) == EquatableCardObject(rank: 5))
// true

print(EquatableCardObject(rank: 5) == EquatableCardObject(rank: 3))
// false

And not just for single values, but for arrays as well.

let equatableCardObjects1 = [EquatableCardObject(rank: 1), EquatableCardObject(rank: 2), EquatableCardObject(rank: 3)]
let equatableCardObjects2 = [EquatableCardObject(rank: 1), EquatableCardObject(rank: 2), EquatableCardObject(rank: 3)]
print(equatableCardObjects1 == equatableCardObjects2)
// true

Summary

This sure looks like a breaking change from 1.2 to 2.0. I wonder if that’s intentional or not. It does look strange that writing implementation of == operator doesn’t actually change anything, in fact, it is never even called for NSObject subclasses, but then the whole conformance to Equatable is broken.

JIRA ID in Git Commit Messages

2015-06-04T00:00:00+00:00

A quick tip on how to automatically add JIRA ID in each git commit message.

What & Why

JIRA or not, if you use some kind of issue tracker and want to get a certain level of integration with Git, it is a good idea to include ticket IDs into commit message.

A couple of assumptions then.

Ticket IDs are in the PROJ-1234 format, where PROJ is the short reference to the project (usually uppercase) at least 1 character long, and 1234 represents a ticket number.

Let’s also assume that you are naming your branches in the following way: prefix/JIRA-1234-optional-description. Prefix is arbitrary and optional as well. Usual examples are feature/ or bugfix/. So now you are working on the branch named like this and you want all your commits to have [JIRA-123] added at the start of commit message, for example

[JIRA-123] Fix the crash

How

You should use Git hooks to achieve the goal.

There are subtle differences when doing it for a main git repo and for submodules.

Main Repo

Git hooks are located in .git/hooks/. The particular hook you are interested in is commit message hook. Copy a default sample .git/hooks/commit-msg.sample and name it .git/hooks/commit-msg.

cp .git/hooks/commit-msg.sample .git/hooks/commit-msg

Now edit commit-msg and remove the example code from it.

# Remove this code.
test "" = "$(grep '^Signed-off-by: ' "$1" |
     sort | uniq -c | sed -e '/^[   ]*1[    ]/d')" || {
    echo >&2 Duplicate Signed-off-by lines.
    exit 1
}

Then add this code at the end of the file.

# If commit message is a fixup message, ignore it.
[[ -n "$(cat $1 | grep 'fixup!')" ]] && FIXUP="YES"

TICKET=$(git symbolic-ref HEAD | rev | cut -d/ -f1 | rev | grep -o -E "[A-Z]+-[0-9]+")
if [[ -n "${TICKET}" && -z "${FIXUP}" ]]; then
    sed -i.bak -e "1s/^/[${TICKET}] /" $1
fi

Some explanation might be helpful. First the line that sets the TICKET variable.

TICKET=$(git symbolic-ref HEAD | rev | cut -d/ -f1 | rev | grep -o -E "[A-Z]+-[0-9]+" | head -n1)

The git symbolic-ref HEAD commands gets the name of the current git branch, which may look like refs/heads/feature/JIRA-1234-description.

Then rev turns it into noitpircsed-4321-ARIJ/erutaef/sdaeh/sfer.

When the branch name is reversed, the last component is now first in the string, and that’s exactly the component we are after, so get it with cut command using / as a separator. cut -d/ -f1 splits by / and takes 1st component. The result is noitpircsed-4321-ARIJ.

Next reverse it back into JIRA-1234-description and at this moment we expect the string to start with JIRA ID. JIRA ID by itself is more than one capital letter, then dash - and then more than one digit, this is exactly what [A-Z]+-[0-9]+ regex describes. If your project name is different, you can adjust regex. grep with the given regex and -o option will return the match only, which is JIRA-1234. Finally head -n1 will take only the first match, that’s in case you have more than one JIRA ID in the branch name.

If the TICKET has a value, then modify current commit message by adding value of TICKET at the start of it. The commit message is actually a file and that file can be accessed via bash script $1 variable. sed is used to find a beginning of the line ^ and add [${TICKET}] , but only once, which is controlled by 1 before before the s/ path: 1s/^/[${TICKET}] /.

if [[ -n "${TICKET}" && -z "${FIXUP}" ]]; then
    sed -i.bak -e "1s/^/[${TICKET}] /" $1
fi

And last thing to explain is FIXUP variable. If you are using commands like git commit -a --fixup HEAD then fixup! string is added to the start of commit message automatically. But the problem is that fixup! is added before commit hook is called, so you end up with commit messages like this.

[JIRA-1234] fixup! [JIRA-1234] Commit message

That’s not what you want if you want to take advantage of autosquasing feature. This is why there is this bit of code at the start of the script.

[[ -n "$(grep 'fixup!' $1)" ]] && FIXUP="YES"

It searches for fixup! string in commit message file $1 and if a match is found sets FIXUP variable to "YES".

Submodules

OK, so now you have commit hook working for main repo, but what if you also have a number of in-house submodules and work with them in the same workspace? In that case you’d want to have the same hooks for submodules.

The problem is that hooks for submodules are located in different directories. Submodules git configuration and other files are located in .git/modules. The simplest way would be to find all hooks directories inside .git/modules and copy commit message hook file we created previously. This will works if you are OK to have same hooks for main repo and submodules, and none of the submodules paths contains hooks string. The script below does exactly what I described in English just now. smh here stands for submodule hooks.

for smh in $(find .git/modules -name hooks); do \
    cp -f .git/hooks/commit-msg ${smh}/; \
done

Don’t forget to update submodule hooks when you change the main repo hooks.

App Store Version Territory

2015-06-01T00:00:00+00:00

Homegrown research on iOS app short and bundle version strings and how to specify them properly in respect to iTunes Connect and TestFlight.

Version Strings Intro

A typical iOS app needs 2 version strings to be defined. This is an example of Xcode 6.3.2 UI.

The first version string is referred to as just “Version”. It is also know as Short Bundle Version String and is stored under CFBundleShortVersionString key in the app’s Info.plist.

This is what your users expect to see on the “About” screen in the app, if they care. The general rule is to use Semantic Versioning scheme in the Major.Minor.Patch form, e.g. 1.0.0. It is also possible to use more user-friendly lightweight option Major.Minor, for example 1.0, or even use just 1, the decision is up to you. Later in this post and in the Summary section, I will give you the exact guidelines for short version string.

Second is Bundle Version. In Xcode UI it is called just “Build” and is stored under CFBundleVersion key in Info.plist. Another common name for this version string is “Build Version” or even “Build Number”. Its purpose is to uniquely identify this particular bundle (aka build) among all the other builds for the current app version. Often bundle version can be unique across all versions of the app ever, though this is not mandatory any more.

You are free to put whatever you want as version strings, it will be compiled, linked and archived with no errors or warnings of any kind. Here’s the ultimate proof for my words.

The real fun begins when you try to upload that app bundle to TestFlight. At this time it actually matters what you put as app version strings. Let’s dissect each version string separately and figure out what the rules are. But first let’s create a test app in iTunes Connect.

Short Bundle Version String

So what happens if I try to upload an app bundle with short version string saying “poop”? Naturally it fails.

I was totally expecting arbitrary strings to be a “no no” in iTunes Connect. But guess what? Big surprise!

Well, don’t do that anyway.

Let’s use a valid version string this time, e.g. 1.0.0 and for now set build version to 1. I will use the short-version (build-version) format from this moment on to indicate both versions, so I’m trying to upload 1.0.0 (5). Why 5? Well, I just want it to be 5, it doesn’t really matter.

Great, so it worked. Note that the app short version string exactly matches the version string in iTunes Connect. Back in a while, before Apple integrated TestFlight into iTunes Connect, it used to be a mandatory requirement, but not any more. I’m sure you want a proof, so I’ll just upload 1.0 (1) this time.

It worked! But that’s not convincing enough. What if there’s some advanced matching happening and 1.0 is kind of the same as 1.0.0? OK then, I will upload 2.0 (1).

Worked again! Just to make sure I’m not uploading to a black hole, I’ll try 1.0.0 (0) again. Note the 0 build version, that’s on purpose.

Aha! A very interesting error message! I personally like the train version term. Apparently each distinct version in TestFlight has its own train, whatever that means, I like to imagine an actual Illawarra line train with carriages. Every time you upload a new prerelease version, a new train is created. The order in which you upload the builds does not matter, you can upload 1.0 after 2.0.

However, if your app already has a live version in App Store, then your new uploads must have a greater version that the live app. Greater in this case means semantically greater, that means components of the version string are compared left to right. For example, 1.0.1 is less than 1.1, because 1 == 1 and 0 < 1. Here’s the proof.

Finally, to dot all the is, or should I say to dot all the short version string components, I will try to upload 2.1.1.1 (1) to see what happens.

And here’s the rule for creating a short version string.

A period-separated list of at most three non-negative integers.

Very clear and simple to remember.

One more thing, before I move on to bundle version. You can upload new prerelease builds to iTunes Connect even if you don’t have a new version created there yet. That’s right, just upload the builds and they will show up in Prerelease tab anyway. That also explains why the version string of uploaded bundle doesn’t have to match the version with Prepare for Submission status. As usual, I can prove it.

OK, enough of short version string. Don’t worry if facts look a bit scattered around the text, I’ll summarize them all in the end. Now it’s time for…

Bundle Version

The rules for bundle version are not the same as for short version string. For convenience and to reduce tautology in the text, I will call it build version by default.

There are many ways to compose a build version. The most popular options are

Integer value (Build Number)
Period-separated non negative integers
Date string in YYYYMMDD or similar format
Git commit hash
Some weird stuff

Let’s look at each option in detail.

Build Number

This is the most basic and simple way to manage a bundle/build version. An incremental non-negative integer. Any CI server gives you this number in some way. The incremental part really matters. As I demonstrated before by uploading 1.0.0 (0) after 1.0.0 (5), the build version of each new upload must be greater than the previous build version on the same train. This time there is no special meaning for greater, just good old integer comparison, so 11 is less than 101.

Another important note is that build versions need to increment in the scope of the same train only. In the past, build version had to be incremental across all versions. So if you had 1.0 (10) live and then tried to upload 2.0 (1) it would fail because 1 < 10. Thankfully that’s not the case any more.

So to sum it up, build number used as bundle version

Must be incremental
Independent for each short version string train

Period-separated

This is a little less popular way to manage build version, but still can be found “in the wild” so to say. For example, a build version is composed by joining short version string with the build number, like major.minor.patch.build: 1.0.0.1, 2.0.42 and so on.

So what happens with this build version string when it gets to iTunes Connect? I will first tell you what happens, then will prove it with concrete examples.

The build version is separated into components using .s as separators
Each component is stripped of leading zeroes
Processed components are joined back together to form one big integer

The resulting integer is non negative, to be more specific it’s unsigned long integer.

Some examples.

1.0.0.0 -> 1, 0, 0, 0 -> 1000
2.001.030 -> 2, 001, 003 -> 2, 1, 30 -> 2130
1.000.02.03.04 -> 1, 000, 02, 03, 04 -> 1, 0, 2, 3, 4 -> 10234

As you can see in the last example, the number of period-separated components is not limited this time. At least I tried all the way up to 4, can’t tell what happens when you have 5 or more of them, but I am about 99% sure the sky is the limit, I mean there’s some limit on the maximum length of the string.

Once the build version string is converted into an integer, the same rules apply as for the simple incremental build number. The newly uploaded build version for the given train must be greater than any previous build version.

Some examples again.

2 < 1.1 [2 < 11]
1.0 < 1.0.0 [10 < 100]
1.0 == 1.01 [10 == 10]
1.0.1.020 == 1.001.1.20 [10120 == 10120]

When you try to upload the same version more than once, you get a “Redundant Binary Upload” error message.

I think I can stop now, the rules are pretty clear.

Date String

Date string is another way to compose a build version. For example, when using YYYYMMDD format your build version may look like 20150604. Actually, this is a very good option. The resulting integer will be always incremental, thanks to the way the world around us works.

If you have an advanced CI setup, you should think about adding more components to the date format. Such as hours and minutes YYYYMMDDHHmm. This way if you have more than one build happening within the same hour you will be able to tell them apart.

If you choose to go with this option, don’t use any kind of separators, such as periods, colons or alike. With periods you will have unwanted side effects when leading zeroes are removed from components. Stuff like colons falls into Weird Stuff category, which I will describe later on.

Git Commit Hash

Another quite popular approach is to use git commit hash as build version string, or as a part of build version string. One useful property of commit hashes is uniqueness. But that’s not enough for TestFlight.

I tried quite a few things, such as hex numbers like a, f, e, e4, 5e and even real life git commit hashes. So I lay the evidence before your eyes.

Obviously, going outside the a - f range will take you nowhere, the poop example in the beginning of the article covers that well, and I tried things like xyz too.

Weird Stuff

Finally, the weird stuff. I’ll just list some of the things I tried.

1+0-2/5
1+++0---2***5
1...0...2...20

Some error messages from iTunes Connect.

Surprisingly though, despite the fact that upload fails, validation is successful. Whatever validation does it doesn’t check the validity of version strings.

Legacy

I subjectively call things like HockeyApp a legacy in this article. You may have different opinion. What I really want to say is that all these iTunes Connect and TestFlight rules do not apply to things like HockeyApp or in-house Over The Air distribution. Despite that fact, it would be reasonable to start making changes towards TestFlight guidelines.

Summary

In conclusion I will make an attempt to summarize the version string rules.

Short Bundle Version String

A period-separated list of at most three non-negative integers
Must be semantically incremental
Must be greater than current live version

Bundle Version

Converted to unsigned long integer
Periods are removed
Leading zeroes removed from each period-separated component
Resulting integer must be incremental
- But only among other bundle versions for the same short version string (version train)
Alphabet characters and git hashes in particular will not work
Various punctuation marks will not work

Of course, I’m not claiming that this is the complete guide. I don’t know what is the limit for maximum version string length. Could be that some punctuation marks are legal, I didn’t try them all. Obviously I am giving no reference to any official documentation, could not find it or didn’t look hard enough. In any case, this information should be enough for you to come up with a robust and reliable versioning strategy for your apps.

Jenkins Job DSL - Configure Block

2015-05-30T00:00:00+00:00

A hands-on experience with Configure Blocks in Jenkins Job DSL.

Jenkins Job SDL is a crown jewel of all the Jenkins plugins. I had a basic write up about it already, and another one in regards to DSL script properties.

This post is an example of using Configure Blocks. I’ll make an effort an try to explain what Configure Blocks are with my own words.

Basically, the whole DSL language output is a job configuration XML file. You are given a high level API to shape out that XML, that’s it. In some cases, where there’s no simple high level API available yet, you can use a bit lower level methods to mess around with resulting XML. That’s exactly what configure blocks are - a fallback mechanism to squeeze more juice out of Job DSL plugin.

Here’s an example, a default configuration of Git plugin will not update submodules recursively and will use default 10 minutes timeout for submodules update operation.

Job DSL provides an API to enable recursive submodules update. This is how you can use it in DSL

That was easy. Here’s how the Jenkins job will look like now.

What if you want to change default timeout for submodules operations now? Unfortunately there’s no DSL command for that yet. This is exactly the case where you’d want to use Configure Block. Scanning through Configure Block examples, I have found this one. Looks like it’s quite easy to add new nodes to the job XML, but you have to figure out how to name those nodes first. The best way to find that out is to modify the job via Jenkins UI and then look at the job XML. So open the job configuration page in the browser, type in the 20 minutes custom timeout value and click save.

Now it’s time to locate the XML on file system. All the jobs data is located in JENKINS_HOME, the easiest way to find Jenkins Home location is again via Jenkins UI. Go to Jenkins > Manage Jenkins > Configure System and have a look at the top of the page.

Navigate to Jenkins Home, then to jobs directory, then find the directory of your particular job, this is very straightforward task, finally you will see config.xml in that directory, open it and have a look. This is the node you are looking for

Specifically you are interested in this block of code

See that <timeout>20</timeout> node? This is exactly what you want to generate using Configure Block. This is how you can do that

That looks easy, doesn’t it? But hold on a sec before you celebrate. Run a DSL script first then have a look at resulting XML before looking at job configuration via Jenkins UI. Locate the SCM node again

Hm… Looks a bit strange doesn’t it? The disableSubmodules and recursiveSubmodules nodes are outside of hudson.plugins.git.extensions.impl.SubmoduleOption for some reason. At least the values look right. So now go ahead and open the job configuration in Jenkins UI or hit refresh if you had it open all this time.

Wait… What did just happen there? The recursive update checkbox is cleared!

Let’s have a look at job XML one more time.

Well, the scm node is moved up in the XML and is changed a bit. Also, the recursiveSubmodules as well as disableSubmodules are moved inside the hudson.plugins.git.extensions.impl.SubmoduleOption node, but somehow recursiveSubmodules has lost its custom value in the process. Whatever Jenkins is doing under the hood is a mystery for me. This may be a bug in Jenkins Job DSL plugin or a known limitation. Brief search in project JIRA and discussion group didn’t come back with any results. That means if you are working with Configure Blocks, keep the following in mind

When changing one particular node, either do all changes via Job DSL API or via Configure Block, but never mix the two.

If you want to have both recursive submodule update and custom timeout configured, set both of them via Configure Block.

Fix Objective-C Imports

2015-05-29T00:00:00+00:00

A way to change Objective-C #import <Framework/Framework.h> to modern @import Framework;.

Modules Syntax

Objective-C modules were first introduced with iOS 7. Modules came to live for a reason, for lots of reasons. They are designed to overcome shortcomings of current preprocessor, specifically the way #imports are handled. You can find more information here and here.

If your app minimum deployment target is iOS 7.0, you are free to switch to using modules for all system frameworks, such as UIKit, Foundation and the rest. With iOS 8.0 and support for custom dynamic frameworks, even your own frameworks can be imported as modules, but in the scope of this article we will look at system frameworks only. In terms of syntax your change would look like this

// Old code
#import <UIKit/UIKit.h>
#import <Foundation/Foundation.h>
#import <UIKit/UIImage.h>       // Just an example

// New code
@import UIKit;
@import Foundation;
@import UIKit.UIImage;

A totally valid question to ask is “Should I even bother with fixing this?”. And the answer is “No, not necessarily”. As soon as you enabled modules for your project all #include and #import statements are automatically mapped to corresponding @import statement. This is described in some detail here and here.

So, technically, fixing imports in legacy code is the question of code base maintenance and code style. It does look a bit off when both #import and @import are sitting few lines apart in the same file. Weigh all pros and cons before making a final decision. Keep in mind the fact that if you already have the script at hand, the conversion will take no time at all.

Fixing Recipe

Xcode is know to offer code migration features, such as “Convert to ARC” or “Convert to Modern Objective-C Syntax”. However, converting all #imports to @imports is not part of Xcode feature set. I would add “yet”, because I believe sooner or later Apple will proclaim iOS 7.0 as minimum OS supported by Xcode and force all imports conversion. Until that happens we need to handle this migration ourselves.

So what would be the recipe to fix imports for all system frameworks? Here’s one I have in mind

Get a list of all system frameworks
For each framework
- Replace #import <Framework/Framework.h> with @import Framework;
- Replace #import <Framework/Header.h> with @import Framework.Header;

System Frameworks

Let’s solve the first part and get a list of system frameworks. Naturally, I googled first and ended up here. My desire to scrape the web page luckily died off right away, reading through the first paragraph I found out that all system frameworks are located in

<Xcode.app>/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/<iOS_SDK>/System/Library/Frameworks

That path looked a bit familiar to me and I knew I could get at least some of it with help of xcrun, so this is what I got in the end

xcrun --sdk iphoneos --show-sdk-path

Then just append the missing bit and list the directory. Everything that ends with .framework is (quite logically) a system framework.

ls $(xcrun --sdk iphoneos --show-sdk-path)/System/Library/Frameworks | grep .framework

Now I would like to turn this into a regular expression. What I want to have is a regex that says something like UIKit OR Foundation OR iAd and so on, with POSIX regex syntax it will look like this UIKit|Foundation|iAd. So I loop through results of ls, drop the .framework bit and create a string of or-ed framework names. My bash skills do not shine really bright in this example, but the code does the job.

# Get the list of all system frameworks
LIBS=$(ls $(xcrun --sdk iphoneos --show-sdk-path)/System/Library/Frameworks | grep .framework)

# Collect all library components and combine into or-ed regex
for lib in ${LIBS}; do
    BASE_NAME=${lib/.framework/}
    [[ -z "${LIB_COMPONENTS}" ]] \
        && LIB_COMPONENTS="${BASE_NAME}" \
        || LIB_COMPONENTS="${LIB_COMPONENTS}|${BASE_NAME}"
done

echo "${LIB_COMPONENTS}"

The output is like this

AVFoundation|AVKit|Accelerate|Accounts|AdSupport|AddressBook|AddressBookUI|AssetsLibrary|AudioToolbox|AudioUnit|CFNetwork|CloudKit|CoreAudio|CoreAudioKit|CoreAuthentication|CoreBluetooth|CoreData|CoreFoundation|CoreGraphics|CoreImage|CoreLocation|CoreMIDI|CoreMedia|CoreMotion|CoreTelephony|CoreText|CoreVideo|EventKit|EventKitUI|ExternalAccessory|Foundation|GLKit|GSS|GameController|GameKit|HealthKit|HomeKit|IOKit|ImageIO|JavaScriptCore|LocalAuthentication|MapKit|MediaAccessibility|MediaPlayer|MediaToolbox|MessageUI|Metal|MobileCoreServices|MultipeerConnectivity|NetworkExtension|NewsstandKit|NotificationCenter|OpenAL|OpenGLES|PassKit|Photos|PhotosUI|PushKit|QuartzCore|QuickLook|SafariServices|SceneKit|Security|Social|SpriteKit|StoreKit|SystemConfiguration|Twitter|UIKit|VideoToolbox|WatchKit|WebKit|iAd

With this regex pattern it is now possible to fix the imports. For the live of me I couldn’t win the battle with sed or awk and use regex back-reference in the pattern itself. This is not to say anything bad about sed or awk as such, this is just me lacking the skills. However, I still managed to stay purely within a realm of Unix-based system by using perl.

Framework/Framework.h

This is exactly the case where back reference needs to be used in the pattern expression itself, not in the replacement. This is an illustration to explain what I actually mean

# Use back-reference in match pattern
"s,(group-capture)/\1,replacement,"

# Use back-reference in replacement
"s,(group-capture),\1,"

The () parenthesis are capturing a group and then captured content can be back-referenced using \1, \2 and so on. In this example (group-capture)/\1 used in pattern match literally means string “group-capture” repeated 2 times and separated by /: “group-capture/group-capture”.

Anyway, assuming the source file name is stored in FILE_PATH variable, the in-place replacement with Perl looks like this

FILE_PATH=SourceFile.m

# LIB_COMPONENTS are set previously
perl -pi -e "s/#import[ \t]*<(${LIB_COMPONENTS})\/\1.h>/\@import \1;/" "${FILE_PATH}"

Here you can see that regex accounts for missing or varying number of spaces after #import and that \1 back-reference works perfect with perl.

Framework/Header.h

This bit is almost the same as the previous case. In fact, regex is a little bit more common and doesn’t use back-reference in match patter, instead uses 2 back-references in replacement string. It is important to note that these perl commands must be applied exactly in the order they are described, so you don’t end up with @import Framework.Framework;.

FILE_PATH=SourceFile.m

# LIB_COMPONENTS are set previously
perl -pi -e "s/#import[ \t]*<(${LIB_COMPONENTS})\/(.*).h>/\@import \1.\2;/" "${FILE_PATH}"

Other Modules

In fact, you know of course there are other modules as well, such as runtime modules, e.g.

// Old style
#import <objc/runtime.h>

// Module
@import ObjectiveC.runtime;

This case is somewhat more complicated than fixing system frameworks. I played a bit with --show-sdk-platform-path option of xcrun command and tried other things, but overall there’s no straightforward obvious solution to this problem. You may have to fix these imports by hand or wait until Xcode automates it.

Summary

As a traditional TL;DR; part, I will just post the link to the upgraded shell script version.

Fix Objective-C File Header Comment

2015-05-29T00:00:00+00:00

A simple way to fix file headers in your Objective-C (an not only) files.

Might be worth to clean up confusions, if any. By “file header” in this case I mean the C-style header comment like this

//
// Copyright (c) 2015 NSBogan. All rights reserved.
//

Problem

For some people it doesn’t really matter what sits at the top of the file. In big projects it may become more important. I have one particular case in mind where information in file headers needs to be anonymous and should contain only a company name and copyright notice, just like the code sample at the top. There are pros and cons to this approach. You may argue that it is important to know who created the file to point a finger, but when the project is large enough, dozens of people end up touching the same file, the project outlives developers (they just come and go, but the project stays), there’s no point to point a finger any more (apologies for the tautology here).

Another note is that we are here talking about existing files. When it comes to creating new files, it is possible to modify default Xcode templates, and even manage to preserve the changes on Xcode upgrade. There’s even a tool to automate the process to a certain level.

However, if you have a script that can fix header comments in existing files, you can reuse the very same script in your git commit hooks to fix the header comments of new files. Kill two birds with one stone (though in my native language we choose hares as objects of violence for this proverb).

Solution

So, at the high level, the way to approach this problem is

Remove all blank lines at the start of the file
Remove all lines that start with // from the start of the file
Insert you custom header comment at the start of the file

Before proceeding further, let’s agree we are working with a source file named Source.m and have an environment variable defined as FILE_PATH=Source.m.

Remove Blank Lines

The blank lines at the start of the file are a rare exception, but sometimes they do occur. Fixing this is possible with sed is easy

sed -i . '/./,$!d' "${FILE_PATH}"

This is one of those they call read-only language. A one-liner that takes hours to decipher, unless you are fluent with sed already.

The -i . option means inplace edit and weird looking . with a must-have space after -i option is OS X specific way to tell sed not to create backup file.
The /./,$ construction is a range and means from /./ to $. /./ is a regex that matches any non-blank line, $ is a special address and means last line of the input. So /./,$ in English means from first non-blank line to the end of file.
! means negation. So if you negate /./,$ you get the following human language description from first line of the input to last blank line inclusive. In other words, all consequent leading blank lines in the file.
Finally, d command means delete, so you tell sed to delete all lines in the given range.

Whoa, that was a lot of words to explain one 7-characters sed command…

Remove Header

Next thing to do is to remove existing header comment. This is a matter of removing consequent lines starting with // from the start of the file. You may think this sounds just like removing consequent blank lines. It does! But I failed big time to do it with sed. I originally assumed this is just a matter of replacing /./ regex with something like /^\/\//, but no, it didn’t work for me. I spent a fair amount of time googling and trying my best with sed and awk, before I gave up and came up with the solution you will see below. However, it would be fair to say that my seding and awking skills are nowhere near good enough, I am sure there’s a better way to do it and I would welcome any comments.

# Count number of consequent lines starting with // at the beginning of the file
N=$(cat "${FILE_PATH}" | awk '{ if(/^\/\//) print; else exit; }' | wc -l | xargs)

# Remove first N lines from the file
[[ ${N} -gt 0 ]] && sed -i . "1,${N}d" "${FILE_PATH}"

You can see here that the very same /^\/\// regex I tried to use with sed works perfectly with awk. The awk command will execute print until file lines match the pattern and will stop (using exit command) as soon as the first non-matching line occurs.

Word count (wc) command is used to count lines then (-l option) and xargs is handy in this case because it strips leading and trailing spaces from the input.

Next if N is greater than zero, use good old sed to remove first N lines from the file. This is done by using 1,${N} address range and delete command. Note the use of “weak” quotation marks in sed regex, this is to resolve reference to N inside the regex string.

Insert New Header

Finally, you have the file stripped of its old header comment, time to insert a new header.

# Insert new header using current year
HEADER="//\\
// Copyright (c) $(date +'%Y') NSBogan All rights reserved.\\
//\\
\\
"

sed -i . "1s|^|${HEADER}|g" "${FILE_PATH}"

The header here is using date command to insert current year in YYYY format, e.g. 2015.

sed matches the beginning of the line (^ anchor) and inserts the header there. The 1 before replacement pattern is there to make sure replacement is made only once.

Edge Cases

The described approach will not work for header comments that use block style, e.g.

/**
    File header comment.
*/

Summary

As usual I link a version of slightly improved script. As a bonus this script can fix header comments in xcconfig files as well. I still say slightly, because it can be improved more. One possible step up is to create a text file with you custom header comment and pass this file path as an input to a header fixing script.

You can bundle this script up with imports fixing script and use as part of git commit hooks to eliminate some of the code style debates during code reviews.

Jenkins Job DSL - Properties

2015-05-17T00:00:00+00:00

Learn how to work with global properties in DSL Groovy script, and how to handle missing properties.

If you are not familiar with Jenkins Job DSL, this post may be a good starting point;

Properties

When you are working with main Groovy script, all Jenkins environment variables are available as so called properties. Properties, because you main script is more than just a similar shell script, in fact you are working with an instance of a Groovy class, that’s why there are properties. For example, you can read the value of Jenkins’ BUILD_NUMBER environment variable in DSL script like this

Very convenient way to read any environment variable. Additionally you can use EnvInject plugin and inject more environment variables to the build job.

Missing Properties

But what happens if environment variable is not defined? In that case corresponding property will not be defined as well, so if you try to read that property, you will get a runtime error.

So what do you do if you really want to read the variable first and if it’s not there just use sensible default? For example

First thing you may try is to use hasProperty method which is a member of any Groovy class.

This code will not throw a runtime error. The problem is that hasProperty will always return false.

Bindings

All is not lost though. Each Jenkins DSL script has a reference to so called “bindings”. Apparently, bindings are used to bind all the environment variables and make them available for the script. Additionally bindings include references to things like standard output, which will come handy later on.

That’s all you have to do to get the script bindings. Now you can access environment variables in a safe way.

If the property is not defined, the return value is null and your code will fallback to a default value.

Standard Output

As I mentioned before, bindings also bind such things as standard output. By default println statements will work only when called from the body of the main script. If you use packages and other classes in your DSL, logging from other modules via println statement will not work, will not yield any output to console to be more specific.

Of course it makes no sense to declare Logger class in the body of the main script. Proper way is to factor it out into a separate file and import the package instead. I plan to have a write up on advanced DSL with classes and packages.

Provisioning Profiles, Sigh...

2015-05-15T00:00:00+00:00

In this article you will find out why iOS provisioning profiles are a nightmare and if there is a way to automate provisioning profiles management.

Provisioning Profiles

Anyone who calls themselves an iOS Developer, knows what this is all about. You have to generate profiles, make sure they use proper certificates and are configured for the right app identifier. Should you renew a certificate or change the app’s entitlements, you now have to regenerate all related provisioning profiles, then make sure you install new profiles everywhere you need them. If it wasn’t difficult enough already, there are 2 types of provisioning profiles: Development and Distribution; and if that wasn’t enough, Distribution profiles come in two more flavors: App Store and AdHoc.

But that’s not the end of the nightmare either! What if you are a member of more than one iOS developer program? For example, you are part of both App Store and Enterprise programs, or you are a freelancer and happen to be a member of a dozen teams. Did you use different Apple IDs for different teams? Doesn’t make it any easier at all. I’m sure many of you know what it feels like.

Finally, things go wild if you have to manage that provisioning zoo for Continuous Integration setup. Just one build box is painful enough to keep up to date, I’m not even mentioning multiple build agents yet…

Of course I’m over pessimistic, but that’s on purpose. One could say that Xcode can manage your accounts and profiles. That’s true but Xcode is known to do a very sloppy job in that regards, and in no way Xcode’s provisioning profile management can be automated.

Overall, managing provisioning profiles involves a lot of manual interactions with web browser. What if there was a tool that would let you manage profiles automatically, via command line and scripts? Well, it so happens that there is one now! Meet Sigh a part of Fastlane tools. Fastlane is a collection of tools that cover all aspects of iOS development, Sigh in particular is designed to help you with managing provisioning profiles and certificates. You can manage profiles installed locally as well as create, renew, update, regenerate and delete profiles from developer portal.

There are so many uses for the tool, but in this article I’ll describe the most basic one, that is downloading an existing provisioning profile. As a bonus, you will see how to parse provisioning profiles and get all the details from it, especially profile UUID and signing identity of the related certificate.

Download Profile

Time to fix all the assumptions before going forward.

You have an account with Enterprise development program and that account has Admin level of access.
You have a Distribution provisioning profile created for Wildcard App ID (*).

That will do it. First thing to do is to install Fastlane Ruby gem.

gem install fastlane

Time to get a profile now.

So what’s going on here?

--username is the Apple ID you used to join the developer program.
--team_id is the 10 character ID of the team. This is mandatory if the Apple ID is part of multiple teams (aka programs). In general, it’s just a good rule of thumb to be as explicit as possible.
--provisioning_name is the name of provisioning profile. That’s the very same name you see when you login to developer portal, not UUID or any other kind of identifier.
--app_identifier is the App identifier in the provisioning profile. The same thing you use as Bundle ID most of the time. In this example the profile is a wildcard profile, so app identifier is just “*” (wildcard). Be cautious! Those quotes around * really matter!
--filename tell sigh where to save a copy of the provisioning profile file. By default it will be installed in ~/Library/MobileDevice/Provisioning Profiles/. If you don’t want the profile to be installed, add --skip_install option to the command.

The very first time you use a given combination of Apple ID and Team ID you will be asked for a password. Fastlane will safely store your password in keychain and you will not have to type it in anymore (well, until the moment you change it, of course). So for CI setup you will have to use Sigh (or any other tool from Fastlane) at least once for each developer account, to have the password stored, and from that moment on all CI scripts will be able to work without user intervention required.

The above is not the case with the introduction of 2FA. You would have to configure FASTLANE_SESSION as described here.

A side note in regards to Team ID. For some reason the real ID is very obscure and doesn’t match the one you would see in Member Center. Luckily, you can run Sigh with wrong Team ID and it will output the list of all correct IDs.

Parse Profile

Another step towards total automation is to include profile download as part of CI job. Usually all CI jobs expect latest provisioning profiles to be installed on the box. CI is either using UUID explicitly, expects profile to have a specific file name or delegates profile lookup task to Xcode. With Sigh you can simplify this bit immensely. Each CI job will make sure the latest profile is downloaded and installed, and named the way it should be named.

Additionally, you may want to extract information from the profile, such as the profile UUID and the name of the signing identity. Why would you do that? Well, you can build the very same project using various provisioning profiles and signing identities, this is the way you can get different builds for Enterprise, AdHoc and App Store distribution, with different bundle IDs, etc. Usually build scripts are configured to specify the provisioning profile specifier and signing identity explicitly via xcodebuild’s PROVISIONING_PROFILE_SPECIFIER and CODE_SIGN_IDENTITY build settings. This is how you can overwrite default build settings in Xcode project.

While specifying signing identity explicitly is important, specifying profile specifier is something that should be done with a lot of caution. The reason is Today Widget and WatchKit extensions. These extensions are separate targets in Xcode project and by default extension targets are dependencies for main app target. It means when you build a main target, all extensions are built as well. Nothing bad so far. The problem is that each of those extensions has its own bundle ID and, as you would expect, a separate provisioning profile. Now, if you explicitly specify profile specifier when building main app target, this build setting will overwrite build settings for all dependent targets as well. Naturally, the build will fail when trying to build Today Widget extension with bundle ID and entitlements from main app target provisioning profile.

So what do you do? Well, previously I’d say to trust Xcode, but these days I’d recommend to use custom build settings instead. E.g. define build settings in this way:

For main app: PROVISIONING_PROFILE_SPECIFIER = MAIN_APP_PROVISIONING_PROFILE_SPECIFIER
For extension: PROVISIONING_PROFILE_SPECIFIER = EXTENSION_PROVISIONING_PROFILE_SPECIFIER

Now you can control individual provisioning profile specifiers for each target, or even put them in a xcconfig file and use that file to override build settings.

In any case, it’ still helpful to be able to parse the provisioning profile, e.g. if you want to have some post build verification to make sure the correct profile has been used for the build.

Start with dropping all the digital signature stuff and getting just the profile XML (plist) part.

I can’t tell you what exactly all arguments of security command mean, you may find better answers in the Inside Code Signing article, where I learned about this command and other commands you will see in the rest of this post. The result is saved to profile.plist.

Now you have a plist with all the data. Let’s get UUID first. The tool that can work with plists is /usr/libexec/PlistBuddy.

This doesn’t need much explanation. If you run this command you will get the UUID of the profile as an output.

Next is signing identity, which is a bit more difficult. Signing identity name is not stored as plain text in the profile, however, it is part of the certificate. The profile stores all associated certificates in DeveloperCertificates key-value pair. That’s right, there can be more than one certificate associated with one provisioning profile. It’s hard to figure out which of the certificates to pick automatically. The assumption is that all of those certificates are valid and any of them can be used to sign the app bundle. If that’s not the case, you need to tidy up things in your developer portal. Sigh can actually help you to do that from command line as well. Anyway, let’s assume the first certificate is the one you need, time to get it.

Oops, looks rather ugly. Seems like PlistBuddy is doing some base 64 decoding under the hood. You can feed this output to base64 utility to encode it back and see the same data as in plist, but do you really need to? Instead, just feed this unreadable data directly to openssl and ask for the subject field.

The output is something like

You only need to get Common Name (CN) part. At the moment I didn’t find anything better than using sed as you can see below. It extracts the string between “/CN=” and “/OU=”. I don’t know in which flavors certificates come, could be this regex needs adjustments in case the order of fields in subject is different.

Alternatively, you can use this regex

It works with the assumption there are no /s in the common name. I think the message here is

Be reasonable and don’t use =s or /s when naming new certificate.

Summary

So now you have a way to download and parse provisioning profile. Automatically, in one go. With little bit of work you can come up with a proper shell script for the task.

Wit a bit more work you can refactor it to take username, team id and other parameters as input arguments.

Hacker Rank in Swift - Reuse Code

2015-03-17T00:00:00+00:00

[OUTDATED]

Reuse Swift IO code for multiple HackerRank assignments.

If you have read this article, you have probably noticed that code to read from stdin must be copied to each assignment file. Even though you need to copy-paste entire solution to HackerRank web site, this goes against DRY principle. In this post I’ll explain how you can keep all stdin code in one file and use it to run code for assignments.

Start by grabbing Swift code, you can use this file for example. Put it in a file, name it StdIO.swift and put it in the root of your HackerRank folder alongside the makefile from this post (you’ll need that makefile later on).

Swift Module

For next step you should have 2 files. A StdIO.swift file created earlier and, say, solve-me-first.swift for the warmup assignment.

let a: Int = readLn()
let b: Int = readLn()
println(a + b)

If you try to run solve-me-first.swift with makefile created in the article mentioned before, you’ll get nowhere. Swift compiler has no idea where to look for readLn methods, so it can’t interpret this script file alone. We have to build a Swift Module for StdIO.swift and then link it with our main Swift file.

Check out this and this link to find out more about building a Swift module. I’ll just provide a summary here. Once a Swift Module for StdIO.swift is built, it will consist of 3 files:

StdIO.swiftmodule - public interface and definitions. An analogue of header files from Objective-C world.
StdIO.swiftdoc - documentation.
libStdIO.dylib - a shared (aka dynamic) library. That’s actually a binary that has all all your code compiled, much alike dynamic library for C, C++, Objective-C and so on. There’s a way to build a static (.a) library as well.

In general, Swift Module can include more than one Swift file. It just so happens that we have only one. To build a module you need to run this command

# Get location of OS X SDK.
export OSX_SDK=$(xcrun --show-sdk-path --sdk macosx)

# Build StdIO module (use `xcrun swiftc` for OS X 10.9).
swiftc -sdk ${OSX_SDK} \
    -emit-library \
    -emit-module StdIO.swift \
    -module-name StdIO

The options are instructing compiler to emit Swift module and shared library. Compiler also needs to know which SDK to build for. As expected, you should have 3 new files created.

StdIO.swiftdoc
StdIO.swiftmodule
libStdIO.dylib

That’s great, the module is ready. Next step is to link it with your main Swift file and run the code. Before you run any command in the shell, you need to add one more line to solve-me-first.swift file.

# This line is new, import StdIO module
import StdIO

let a: Int = readLn()
let b: Int = readLn()
println(a + b)

Now it’s time to link the two and run the code. This time we can use swift command.

# Get location of OS X SDK
export OSX_SDK=$(xcrun --show-sdk-path --sdk macosx)

cat tc/solve-me-first-tc0.txt | \
    swift \
        -sdk ${OSX_SDK} \
        -l$(pwd)/libStdIO.dylib \
        -I $(pwd) -module-link-name StdIO \
        solve-me-first.swift

Let’s talk about each option separately.

-l<library> - this is a flag that should be followed by the name of the library to link with.
- Actually, -lStdIO works as well, but that’s because libStdIO.dylib is sitting in the same folder. I choose to be more verbose and use the full path to shared library. My intentions will become clear later when we talk about makefiles.
-I <import-path> - this flag is used to specify import path. Similar to include path in Objective-C world, this way we tell the compiler where to look for Swift modules to resolve import statements in the code.
- Once again, instead of specifying current folder as ., I’m using full path. That will be explained later.
-module-link-name <name> - well, it expects a name of the module to link with.

Actually, this is a bit of a cheat. This second command does not compile the code, but interprets it while linking with existing module. There is a way to use Swift compiler here as well, you can find the reference in the end of this post.

OK, so run this command and you should get a (high) 5 as output. Replace the name of the test case file and the name of Swift file and you can run code for any other assignment. But that is too verbose. It should be automated with…

Makefile

… with Makefile, of course. With small effort we can convert shell script into a makefile script.

# Makefile

# Actual directory of this Makefile, not where it is called form
SELF_DIR := $(dir $(lastword $(MAKEFILE_LIST)))

# Build directory
BUILD_DIR := $(CURDIR)/build

# OSX SDK
OSX_SDK := $(shell xcrun --show-sdk-path --sdk macosx)

# Swift executables and StdIO library name
# TODO: 10.9 use 'xcrun swift(c)', since 10.10 can use just 'swift(c)'
SWIFT = swift
SWIFTC = swiftc
SWIFT_STDIO = StdIO
SWIFT_STDIO_LIB_NAME = lib$(SWIFT_STDIO).dylib

# Test cases configuration
TC_DIR = tc
TC = 0
tc-path = $(TC_DIR)/$(patsubst %.$(2),%-tc$(1).txt,$(3))


# Enable phony targets
.PHONY:

# Swift compile and run
%.swift: .PHONY
	@mkdir -p $(BUILD_DIR)

	@# build stdio module
	@$(SWIFTC) \
		-sdk $(OSX_SDK) \
		-emit-library \
		-o $(BUILD_DIR)/$(SWIFT_STDIO_LIB_NAME) \
		-emit-module $(SELF_DIR)/$(SWIFT_STDIO).swift \
		-module-name $(SWIFT_STDIO)

	@# run linking against stdio module
	@cat $(call tc-path,$(TC),swift,$@) \
		| $(SWIFT) \
			-sdk $(OSX_SDK) \
			-l$(BUILD_DIR)/$(SWIFT_STDIO_LIB_NAME) \
			-I $(BUILD_DIR) -module-link-name $(SWIFT_STDIO) \
			$@

OK, so let’s walk the code.

First we declare 3 variables: SELF_DIR, BUILD_DIR and OSX_SDK.

SELF_DIR is an absolute path to the location of the makefile itself. We need to know this information, because we also know that StdIO.swift is located in the same folder as Makefile. It is not the same as $(CURDIR), $(CURDIR) is the folder you run makefile from.
BUILD_DIR is the place to put build output. Don’t forget to put it in your .gitignore and remove it as part of clean target.
OSX_SDK is the path to current Mac OS X SDK.

One important note is the use of := instead of just = when assigning values to the variables. If right side of assignment is one of the makefile functions, like shell to execute shell command, or dir, then each time you reference a variable, for example $(OSX_SDK), the shell command will be executed. If the command is expensive, this will slow down execution of your makefile targets. Using := ensures that variable is assigned a value only when initialized, when it’s reference later on it uses that initial value.

Next block declares group of variables used as a reference to Swift executables, StdIO module and library name.

Then there are 2 variables and 2 functions related to test cases configuration. To get more details about tc-path function, check out previous post. Declaring targets as phony is something that’s explained in that article as well.

Finally, the %.swift target runs all the compile commands. This code is almost identical to the shell code we had before, with minor differences.

-o <library-name> option is used to explicitly tell compiler to put library file and accompanying module files into a build directory. That’s just to keep your workspace clean of compiler output.
file name for -emit-module is referencing $(SELF_DIR) variable to find StdIO.swift in the same folder as Makefile file.

Give it a try and run it.

# Using `make` directly
make -f ../../Makefile solve-me-first.swift

# Using `hrrun` alias
hrrun solve-me-first.swift

hrrun is an alias defined in previous article as well.

So it works now and you can keep stdin code separately and reuse it in assignments source. By now you know how to run code in interpreter and how to compile it, wouldn’t it be nice to be able to choose any of the two options?

Interpret or Compile

Since makefiles support if-else flow control statements, you can add a conditional statement and split your makefile code in 2 parts, one for running code using interpreters, another to compile before running. It’s as easy as this

# Makefile

# Compile (YES) vs interpret (NO)
COMPILE = NO

# Enable phony targets
.PHONY:

ifeq ($(COMPILE), NO)	# Interpret

# Swift
%.swift: .PHONY
	# TODO: Interpret

else	# Compile and run

# Swift compile and run
%.swift: .PHONY
	# TODO: Compile and run

endif

By controlling value of COMPILE variable you can choose one of the two ways to run assignments code. Another alias would be handy as well.

HACKER_RANK_HOME="${HOME}/Projects/hackerrank"
alias hrrun="make -f ${HACKER_RANK_HOME}/Makefile"
alias hrrunc="make -f ${HACKER_RANK_HOME}/Makefile COMPILE=YES"

Summary

You can now do the same thing for Haskell, Python, C, C++, Java or any other language that support compilation. Grab the latest version of Makefile for your reference, and happy HackerRanking!

HackerRank in Swift - Makefiles

2015-03-16T00:00:00+00:00

A basic makefile to help you with testing your HackerRank solutions locally.

So you are into HackerRank and want to be able to run the code locally before submitting it. This is useful when you have one of the test cases failing and you want to debug it and find the problem.

In this article I’ll show how you can run Haskell and Swift code locally via command line in a convenient way.

Manual Mode

Before you automate any process, you need to understand what’s there to automate in the first place. The usual approach is to run all the commands manually, so you can notice common patterns and find out certain edge-cases.

Let’s look at first challenge, which is already solved in this article. Let’s agree that all test cases will be put in tc folder, then create tc/solve-me-first-tc0.txt with the following contents.

2
3

The tc<N> part of the file name stands for “test case N”, the default test case provided as part of assignment has index 0. So to run your code against test cases 0 and 1, you would run commands like these.

# Haskell
cat tc/solve-me-first-tc0.txt | runghc solve-me-first.hs
cat tc/solve-me-first-tc1.txt | runghc solve-me-first.hs

# Swift (use `xcrun swift` for OS X 10.9)
cat tc/solve-me-first-tc0.txt | swift solve-me-first.hs
cat tc/solve-me-first-tc1.txt | swift solve-me-first.hs

Clearly there’s a pattern there, let’s express it using bash syntax.

# Test cases directory
TC_DIR=tc
# Test case number
TC=0
# Base name for assignment
ASSIGNMENT=solve-me-first

# Run Haskell, test case 0

# Interpreter command
CMD=runghc
# File name extension
EXT=hs

cat ${TC_DIR}/${ASSIGNMENT}-tc${TC}.txt | ${CMD} ${ASSIGNMENT}.${EXT}

# Run Swift, test case 1

# Interpreter command
CMD=swift
# File name extension
EXT=swift

TC=1 cat ${TC_DIR}/${ASSIGNMENT}-tc${TC}.txt | ${CMD} ${ASSIGNMENT}.${EXT}

The script above is a valid bash script. With little work you can turn it into a script that takes file name as a argument and does all the magic of figuring out the extension and interpreter command under the hood, but there is a better way (in my humble opinion).

Makefile

Yes, makefiles. I already wrote about them in regards to automating mobile CI tasks. Check that write up if you need to refresh some make and makefile basics. So let’s turn bash script from previous paragraph into a makefile.

# Makefile

# Test cases directory
TC_DIR = tc
# Test case number
TC = 0

# Enable phony targets
.PHONY:

# Haskell
%.hs: .PHONY
	@cat $(TC_DIR)/$(patsubst %.hs,%-tc$(TC).txt,$@) | runghc $@

# Swift
%.swift: .PHONY
	@cat $(TC_DIR)/$(patsubst %.swift,%-tc$(TC).txt,$@) | swift $@

help:
	@echo "Run test case $(TC) for given source file."
	@echo "Use TC variable to specify test case number."
	@echo "Targets:"
	@echo "\t- .hs files for Haskell."
	@echo "\t- .swift files for Swift."

Nice feature or makefiles is pattern matching for targets based on the input. If you run this makefile specifying Haskell source file as input (that means file has .hs extension), it will match agains %.hs target and run commands for Haskell. patsubst is short for “path substitution” and turns input file name into a test case file name, here $@ is the way to refer to target name.

Save it as, say, Makefile and give it a try.

# Run Haskell for default test case (0)
make -f Makefile solve-me-first.hs

# Run Swift for test case 1 (you have to create one)
make -f Makefile solve-me-first.hs TC=1

Works as expected. All you have to do now for each new assignment is to create source file and test case files and then run make. Annoying part is that you have to specify makefile name explicitly each time. Say, you have hackerrank directory with Makefile in it, and you organize assignments in their own directories.

hackerrank
├── Makefile
├── alg
│   ├── arr-n-srt
│   │   └── tc
│   ├── geometry
│   │   └── tc
│   └── warmup
│       └── tc
└── shell

So if your working directory is warmup and you want to run assignment code from there, you have to do somethign like this

make -f ../../Makefile solve-me-first.hs

The solution is to use an alias like this

# in your shell profile (.bash_profile, .zshrc, etc.)
HACKER_RANK_HOME="${HOME}/Projects/hackerrank"
alias hrrun="make -f ${HACKER_RANK_HOME}/Makefile"

Put this alias in your shell profile and now running assignments code from any folder is as easy as

hrrun solve-me-first.hs
hrrun solve-me-first.swift TC=1

Optimize Makefile

Just like with shell scripts, there a bits of reusable code noticeable in the makefile. With use of functions and bit of refactoring, it can be turned into this.

# Test cases directory
TC_DIR = tc
# Test case number
TC = 0

# Get the path to test case file
# usage: $(call tc-path,TC,EXT,FILE)
# where:
# TC - test case number
# EXT - file extension (e.g. hs, swift, rb, py, etc.)
# FILE - name of the source file (use $@ to pass it from recipe)
tc-path = $(TC_DIR)/$(patsubst %.$(2),%-tc$(1).txt,$(3))

# Run test case using interpreter
# usage: $(call run-tc,TC,EXT,FILE,CMD)
# where:
# TC - test case number
# EXT - file extension (e.g. hs, swift, rb, py, etc.)
# FILE - name of the source file (use $@ to pass it from recipe)
# CMD - interpreter command to run the source file (e.g. runghc, xcrun swift, etc.)
run-tc = cat $(call tc-path,$(1),$(2),$(3)) | $(4) $@

# Enable phony targets
.PHONY:

# Haskell
%.hs: .PHONY
	@$(call run-tc,$(TC),hs,$@,runghc)

# Swift
%.swift: .PHONY
	@$(call run-tc,$(TC),swift,$@,swift)

help:
	@echo "Run test case $(TC) for given source file."
	@echo "Use TC variable to specify test case number."
	@echo "Targets:"
	@echo "\t- .hs files for Haskell."
	@echo "\t- .swift files for Swift."

This can be improved and improved until the moment your internal perfectionist is happy, but let’s stop here for now. More important feature of this makefile is how easy it is to add another interpretable language support, for example Ruby.

# Makefile

# ***

# Ruby
%.rb: .PHONY
	@$(call run-tc,$(TC),rb,$@,ruby)

Use slightly modified Ruby solution for Solve me first from HackerRank and try it out.

# solve-me-first.rb
a = gets.to_i
b = gets.to_i
print(a + b)

hrrun solve-me-first.rb

Works like a charm. You can add Python, Go, JavaScript and any other interpretable language in this way. I keep putting stress on interpretable because languages like C or C++ would require to compile and link the code before you could run it, but that’s a story for another post…

HackerRank in Swift - StdIn

2015-03-15T00:00:00+00:00

[OUTDATED]

A way to read standard input for HackerRank assignments in Swift.

HackerRank is an amazing resource. It features lots of programming assignments from multitude of domains. This is a perfect place to prep yourself up for upcoming interview, to improve your problem solving skills and even to learn a new language.

For all those who follows Swift programming language closely, it is now supported by HackerRank. So if you are desperate to write some Swift code and don’t have any real projects to apply it, HackerRank might be an excellent place to do just that.

Well, Swift support with HackerRank has been on and off lately. Few months after I wrote this post Swift 1.2 was released, few days later it disappeared from HackerRank. Probably they have problems catching up with Swift development schedule. In any case you can still solve the problems in Swift and then submit them in bulk once HackerRank engineers get it working.

Most (if not all) of the assignments require reading data from standard input. The format is usually like this.

<N>
<test-case-1>
<test-case-2>
...
<test-case-N>

In this example N is total number of test cases. It is then followed by N lines each corresponding to an input for a test case. This is common, but not the only way to provide input. Some assignments would have 2-line test cases or other format. In any case, one common feature between all assignments is that you need to read data line by line. So let’s start with implementation of core getLine function in Swift.

Get Line

import Foundation

// MARK: Standard Input

/// Reads a line from standard input
///:returns: string form stdin
public func getLine() -> String {
    var buf = String()
    var c = getchar()
    // 10 is ascii code for newline
    while c != EOF && c != 10 {
        buf.append(UnicodeScalar(UInt32(c)))
        c = getchar()
    }
    return buf
}

The code is straightforward.

Declare buf variable used to accumulate the result string.
Declare c variable used to read next character from stdin with getchar
Loop until reach the end of file (EOF), or newline (ASCII code 10)
- On each iteration append newly read character to the accumulator string

You’ve probably heard lots of things about Swift. Among all the things, one very important feature is its interoperability with Objective-C and C languages. This small code snippet isn’t a 100% “pure” Swift code. First of all, it’s using getchar function from C Standard Library made available via Foundation framework import. No worries though, all these APIs are toll-free bridged to Swift. This also explains a chain of initializers / type-casts when appending new character to accumulator string. getchar() returns ASCII code of the character of Int32 type. The thing is that Swift’s String is not your good old ASCII null-teminated C string, it is actually a collection of Unicode scalars. Bear in mind, that collection isn’t just a figure of speech, it actually means that String type conforms to CollectionType protocol. Anyway, to create an instance of UnicodeScalar you need to have a value of UInt32 type, hence the conversion of c to UInt32.

An important note. This code will not work properly with actual Unicode input. Anything outside of standard ASCII table will be rendered as some gibberish. Obviously getchar is not up for the job of reading unicode characters. However, on HackerRank you would never get non-ASCII input (at least for assignments that I saw), so this function does perfect job for its targeted application area.

Read Line

Now when you can get a line, next thing you will want is to convert that line into something else. For example, if the line is just one integer, you would like to convert into a value of Int type. It it’s a list of integers (space- or comma-, or whatever- separated), you would obviously want to convert it to Array<Int> or [Int]. And so on and so forth.

I am actually inspired by Haskell here. It has a core getLine function to get a line as string, then it also has readLn method, which doesn’t just get the line, but also allows converting it into a value of desired type.

-- Haskell

-- Read a line and convert to IO Int
n <- readLn :: IO Int

-- Get a line and convert (read) it as array of Int
line <- getLine
let input = read line :: [Int]

So I wanted to have something similar or alike to use in Swift. This is one of solutions.

/// Read line from standard input
///:returns: string
public func readLn() -> String {
    return getLine()
}

/// Read line from standard input and convert to integer
///:returns: integer value
public func readLn() -> Int {
    return getLine().toInt()!
}

/// Read line and convert to array of strings (words)
///:returns: array of strings
public func readLn() -> [String] {
    return getLine().componentsSeparatedByCharactersInSet(NSCharacterSet.whitespaceCharacterSet())
}

/// Read line and convert to array of integers
///:returns: array of integers
public func readLn() -> [Int] {
    let words: [String] = readLn()
    return words.map { $0.toInt()! }
}

Let’s review each function.

public func readLn() -> String
- This is really just an alias for getLine() as you can clearly see from its implementation.
public func readLn() -> Int
- This function gets the line and converts it to Int using toInt() method. Since toInt() returns an optional, we have to use explicit unwrapping (! operator). Needless to say that if the string can’t be parsed into an integer, the code will crash.
public func readLn() -> [Strings]
- Get the line and parse it into an array of strings. It works from the assumption that default separator is whitespace. Luckily, this is the case with all HackerRank assignments I’ve seen so far. It’s possible to improve this function and pass custom separator string as an argument.
public func readLn() -> [Int]
- Get the line and parse it into an array of integers. You might have thought that this function kind of calls itself. Of course it’s not true and there’s no recursion here. Because the line let words: [String] = readLn() explicitly specifies the type of the words constant, Swift compiler calls the public func readLn() -> [String] function matching the [String] type of words. Once it gets an array of strings, it maps each value to its integer counterpart with toInt() call.

Obviously, you can define as many readLn functions as you wish, but the sane thing to do is to define new function when particular assignment requires it. For example, somewhere down the middle of Warmup section in Algorithms domain, you will need a function like this.

// An array of (Int, String) tuples
public func readLn() -> [(Int, String)] {
     // ...
}

Generics?

OK, so when you see 4 readLn functions that differ by return type only, what does the common sense tell you? And what does each and every functional programming text book recommend in this case? Right - use generics.

But hold on, this is probably the case where generics wouldn’t work right away (I’d be happy to be proven wrong). Generics really work well when lots of functions differ only in types they use, but still have the same implementation details. In this case, each readLn implementation is specific and there aren’t lots of common patterns to be reused. Let’s have a pseudo-code-thought experiment though.

public func readLn<T>() -> T {
     // Not a real Swift code!
     switch T {
     case String:
        return getLine()
     case Int:
        return getLine().toInt()!
     case [String]:
        return getLine().componentsSeparatedByCharactersInSet(NSCharacterSet.whitespaceCharacterSet())
     case [Int]:
        let words: [String] = readLn()
        return words.map { $0.toInt()! }
     default:
        return getLine()
     }
}

Now, this is not a real code and would never compile. The reason is that there’s no simple way to do a switch by the type T. I have found an ugly way to work around this limitation.

public func readLn<T>() -> T {
    if (T.self as? String.Type != nil) {
        return getLine()
    } else if (T.self as? Int.Type != nil) {
        return getLine().toInt()!
    } else if (T.self as? [String].Type != nil) {
        return getLine().componentsSeparatedByCharactersInSet(NSCharacterSet.whitespaceCharacterSet())
    } else if (T.self as? [Int].Type != nil) {
        let words: [String] = readLn()
        return words.map { $0.toInt()! }
    } else {
        return getLine()
    }
}

The conditions for ifs are actually a valid Swift code, though they don’t look pretty. The code still doesn’t compile. The problem is that each return statement returns one of these types: String, Int, [String] or [Int], and compiler doesn’t know how to initialize an instance of type T with any of these types. Compiler doesn’t really know anything about T, but it does know that T gives no guarantee to provide init(_: Int) or any other initializer.

What you could do next, is to try to come up with a protocol which T would conform to. Pack all these initializers inside the protocol, and then much more; and probably this path of generic functional madness would take you somewhere after all. But look at this from another angle: even if you succeed in making this generic function to compile and work, you effectively have implementations of each and every separate functions sitting right there, each in its own if-clause. So what’s the point then?

My personal takeaway from this exercise is: do not overcomplicate things. Just think of a group of N different readLn functions as some single abstract generic function. After all, when you use it in code, that’s exactly how it looks and works.

Use

Finally, after all this abstract talk, it is time to put these standard input functions to a good use.

Let’s start with Solve me first. Here’s the solution in Swift.

let a: Int = readLn()   // calls readLn() -> Int, because a is of Int type
let b: Int = readLn()
println(a + b)

To test it, make a simple solve-me-first-tc0.txt file with test input.

2
3

Then run this command in the terminal.

# Mac OS X 10.9+
cat solve-me-first-tc0.txt | xcrun swift solve-me-first.swift

# Mac OS X 10.10+
cat solve-me-first-tc0.txt | swift solve-me-first.swift

As expected, the output is 5 and you’ve just got your first assignment solved in Swift.

While we are at it, let’s solve the Solve me second. You have to read an array of integers from each line and sum them up. Calculating the sum of all elements in an array is a text book example of using reduce method.

let n: Int = readLn()

for _ in 0..<n {
    let ints: [Int] = readLn()  // calls readLn() -> [Int]
    let sum = ints.reduce(0, +)
    println(sum)
}

Now, if you are one of fresh functional programming converts, you might frown at the for-loop. I had the same feeling originally. Reading a lot about functional approach made me think that any for-loop should be effectively replaced with map, reduce or filter. But is it really so? Swift does have a number of features which are part of functional programming paradigm, but that doesn’t mean that for-loop is now banned from use. For-loop is one of the native Swift idioms and does have its own use where appropriate. As an example, check this post and related discussion on Apple Developers forum.

Summary

You now have a handy stdin Swift library to get you going with most of assignments. If you are not a big fan of copy-pasting these functions each time, check out my other posts about HackerRank makefiles and reusing Swift IO library.

Password Generator in Swift

2015-02-23T00:00:00+00:00

Simple password generator in Swift.

One day I was chatting with other iOS devs and someone posted an example of password generator code. The code featured “for-i-in” loops and other things which didn’t look much swifty to my eye.

I tried to find a real application for nice Swift features like generators, sequences and some functional methods like reduce. I went through a number of iterations to get to the final state. Here’s the code I eventually came up with.

import Foundation

struct PasswordGenerator: SequenceType {
    let length: Int
    let characters: Set<Character>

    func generate() -> GeneratorOf<Character> {
        var currentLength: Int = 0
        return GeneratorOf<Character> {
            let randomIndex = Int(arc4random_uniform(UInt32(count(self.characters))))
            let setIndex = advance(self.characters.startIndex, randomIndex)
            return (currentLength++ < self.length ? self.characters[setIndex] : nil)
          }
    }

    func password() -> String {
        return reduce(self, "") { (pwd: String, char: Character) -> String in
            "".join([pwd, String(char)])
        }
    }
}

let characters = Set("АБВГДAAAAbCd1234567890-=!@#$%^&*()_+QWE˙ÎÓÔ◊ı´¨°•RTYUIOP{}ASDFGHJKL:\"ZXCVBNM<>?")

let passwordGenerator = PasswordGenerator(length: 10, characters: characters)

println(passwordGenerator.password())
println(passwordGenerator.password())
println(passwordGenerator.password())
println(passwordGenerator.password())

As you can see this code is using Swift 1.2 features, such as new Set data type.

The general idea is to create a password generator for a given set of characters to generate passwords of the given length. I’m not claiming here that this is the best design ever. Probably it would be better to have a function that takes a set of characters and password length as an input and then do all heavy lifting in that function.

However, let’s walk through this code.

The PasswordGenerator is a sequence since it conforms to SequenceType protocol. As part of protocol implementation this struct needs to implement generate() function that returns another Swift type Generator. In this example I’m generating characters for a password, so I chose to initialize password generator with a set of characters (characters) and length of the password to be generated (length).

Note that I’m not using any unicode specific code at all. That’s the beauty of String type in Swift. Since String conforms to the very same SequenceType it can easily be converted to an array or a set, and what’s even better, it will be an array or a set of unicode characters.

OK, so back to PasswordGenerator. generate() function returns generator of Character where GeneratorOf is another Swift generic type useful to declare generators of a certain type. Inside the function I’m creating a generator of Character with a closure, this closure captures currentLength variable as well as self. The closure first generates a random index of an element in the set using good old arc4random_uniform. The randomIndex is of type Int and can not be used as a set index. The set index in this case needs to be of type SetIndex<Character>. To get set index I’m using standard library function advance that gives me a randomIndex set index equivalent for a set. Finally I can get a character from the set, incrementing currentLength and comparing it to self.length along the lines to know when to stop. Note the use of self, this is required because it’s a closure.

Last step is to implement password() function that actually generates passwords. This is where I can use the fact that PasswordGenerator is a sequence. Thanks to that I am able to use standard library reduce on self. Reduce iterates through the sequence on each step getting a random element from the characters set it then appends new character to accumulator string using join function.

Without a single for-i-in loop I get a simple password generator that demoes beauty and flexibility of Swift and is yet open for further improvements.

What's your destination?

2015-02-18T00:00:00+00:00

Few hands on tricks about -destination option of xcodebuild.

Destination option (-destination) was a new addition to Xcode 5 release. It is documented on xcodebuild man page and you get the same by running man xcodebuild. This option lets you be more verbose with your xcodebuild commands. For example, if you have a physical device plugged in, running xcodebuild with no destination specifier will build the project for this physical device. Using destination specifier you can explicitly tell xcodebuild to build for simulator.

Allow me just to quote the man page here

The -destination option takes as its argument a destination specifier describing the device (or devices) to use as a destination.  A destination specifier is a single argument consisting of a set of comma-separated key=value pairs.  The -destination option may be specified multiple times to cause xcodebuild to perform the specified action on multiple destinations.

By the way, pay a closer attention to the last sentence. It’s a really nice feature, for example with single command you can run tests on multiple simulators.

One of the keys supported is “platform” and there are 3 options: “OS X”, “iOS” and “iOS Simulator”. Let’s get more details on the last two. iOS and iOS Simulator platforms both require a “name” and “OS” keys to specify device name and iOS version. The question is “How to get list of all names and OS versions?”. There are many ways to do that.

Instruments

The Xcode Instruments command line utility has an undocumented -s option, which lists all devices and templates. By adding devices to the invocation only devices are be listed.

instruments -s devices
# Or
xcrun instruments -s devices

Example output, note that it also includes your Mac as a device.

Known Devices:
R5003398 [5C25B8A6-EDF5-5856-B8B9-CB28DFBFADC4]
Maksym's iPhone (8.1.3) [5dc0e5a8a9f7c0af1feea4bfa13860c9e61350d6]
MyIPhone6 (8.1 Simulator) [9151073B-3B7D-441A-8069-E797FA5059CE]
Resizable iPad (8.1 Simulator) [15FC1628-B369-44DE-8CE0-756439D9AEBC]
Resizable iPhone (8.1 Simulator) [5E7A9E8C-D3A8-4E58-935F-8754B100E867]
iPad 2 (7.1 Simulator) [43DD0C31-685F-4FC8-941F-0BB9B305CD2D]
iPad 2 (8.1 Simulator) [8D722A9F-2DA9-45A2-BAFB-44075AA519D6]
iPad Air (7.1 Simulator) [2FB923E2-F0F4-49DE-8FC3-81960386B8CE]
iPad Air (8.1 Simulator) [6032BE94-1A62-4ADB-9013-50FAB2B088F6]
iPad Retina (7.1 Simulator) [F1967F38-BE03-430F-BFB1-AFF4E2E4511C]
iPad Retina (8.1 Simulator) [E8652514-3042-4958-9027-48A9A95392CE]
iPhone 4s (7.1 Simulator) [57DBC56E-1E04-4384-9A07-7B47C119BEAB]
iPhone 4s (8.1 Simulator) [DDF255C3-8BEF-4A97-8B3F-AD651BABF6CB]
iPhone 5 (7.1 Simulator) [BAF484D4-803C-478D-BAC1-916376C22C48]
iPhone 5 (8.1 Simulator) [45E5CE55-093D-48C0-AA26-AA4E5C569FDB]
iPhone 5s (7.1 Simulator) [6856CCCF-842E-48E8-897B-87BB1865CD10]
iPhone 5s (8.1 Simulator) [0AC6BB28-E860-4320-A532-272BE43C067C]
iPhone 6 (8.1 Simulator) [7953EE9F-19E8-4D0C-9219-268FB0BA6626]
iPhone 6 Plus (8.1 Simulator) [E574C367-1BAA-4F9F-BB92-BD5F6BAB1226]

`simctl`

Another option is to use xcrun simctl, which is a new addition to Xcode 6. In fact, simctl looks like a very promising tool that allows you to create, boot, launch and then shutdown and destroy iOS simulators on the fly, and provides commands to install and launch specific apps. If you ever used Genymotion you might have created Android simulators using VirtualBox command line utility, Apple works towards the same flexibility with simctl.

In context of this article, to see a list of available devices, run this command

xcrun simctl list

You will get the list of device types, runtimes and devices. Device types and runtimes will be handy if you need to create new simulators. You can tell simctl to filter output, for example, list devices only

xcrun simctl list devices

Sample output

== Devices ==
-- iOS 7.0 --
    iPhone 4s (6CF0F409-F4F8-4BCD-AC4E-30E58947A3EB) (Shutdown) (unavailable)
    iPhone 5 (1C08AA1B-9C0C-499B-9570-972EF52941D9) (Shutdown) (unavailable)
    iPhone 5s (3DD4CAA5-6CD7-4DFB-AF95-6494EAE6E90F) (Shutdown) (unavailable)
    iPad 2 (C4E52F2A-C1C0-408F-8177-5AAF7895D3B5) (Shutdown) (unavailable)
    iPad Retina (D145A9C9-DFD0-4564-8A45-BF68A80426AC) (Shutdown) (unavailable)
    iPad Air (091FC037-5C7F-45FC-9089-E424A9806D60) (Shutdown) (unavailable)
-- iOS 7.1 --
    iPhone 4s (57DBC56E-1E04-4384-9A07-7B47C119BEAB) (Shutdown)
    iPhone 5 (BAF484D4-803C-478D-BAC1-916376C22C48) (Shutdown)
    iPhone 5s (6856CCCF-842E-48E8-897B-87BB1865CD10) (Shutdown)
    iPad 2 (43DD0C31-685F-4FC8-941F-0BB9B305CD2D) (Shutdown)
    iPad Retina (F1967F38-BE03-430F-BFB1-AFF4E2E4511C) (Shutdown)
    iPad Air (2FB923E2-F0F4-49DE-8FC3-81960386B8CE) (Shutdown)
-- iOS 8.1 --
    iPhone 4s (DDF255C3-8BEF-4A97-8B3F-AD651BABF6CB) (Shutdown)
    iPhone 5 (45E5CE55-093D-48C0-AA26-AA4E5C569FDB) (Shutdown)
    iPhone 5s (0AC6BB28-E860-4320-A532-272BE43C067C) (Shutdown)
    iPhone 6 Plus (E574C367-1BAA-4F9F-BB92-BD5F6BAB1226) (Shutdown)
    iPhone 6 (7953EE9F-19E8-4D0C-9219-268FB0BA6626) (Shutdown)
    MyIPhone6 (9151073B-3B7D-441A-8069-E797FA5059CE) (Shutdown)
    iPad 2 (8D722A9F-2DA9-45A2-BAFB-44075AA519D6) (Booted)
    iPad Retina (E8652514-3042-4958-9027-48A9A95392CE) (Shutdown)
    iPad Air (6032BE94-1A62-4ADB-9013-50FAB2B088F6) (Shutdown)
    Resizable iPhone (5E7A9E8C-D3A8-4E58-935F-8754B100E867) (Shutdown)
    Resizable iPad (15FC1628-B369-44DE-8CE0-756439D9AEBC) (Shutdown)

To get more details just run xcrun simctl list -h to get help on list command.

`xcodebuild`

Finally, to get the most detailed output, ask xcodebuild itself. It is not really documented and doesn’t look like a proper approach, but it works. The idea is to give xcodebuild invalid key-value pair in destination specifier and wait until it complains about it offering a list of valid options in return.

xcodebuild test -project MyProject.xcodeproj -scheme MyScheme \
    -destination "name=NoSuchName" -destination-timeout 1

The output is

xcodebuild: error: Unable to find a destination matching the provided destination specifier:
    { name:NoSuchName }

  Unsupported device specifier option.
  The device “My Mac” does not support the following options: name
  Please supply only supported device specifier options.

  Available destinations for the "MyScheme" scheme:
    { platform:iOS Simulator, id:C4E52F2A-C1C0-408F-8177-5AAF7895D3B5, OS:7.0, name:iPad 2 }
    { platform:iOS Simulator, id:43DD0C31-685F-4FC8-941F-0BB9B305CD2D, OS:7.1, name:iPad 2 }
    { platform:iOS Simulator, id:8D722A9F-2DA9-45A2-BAFB-44075AA519D6, OS:8.1, name:iPad 2 }
    { platform:iOS Simulator, id:091FC037-5C7F-45FC-9089-E424A9806D60, OS:7.0, name:iPad Air }
    { platform:iOS Simulator, id:2FB923E2-F0F4-49DE-8FC3-81960386B8CE, OS:7.1, name:iPad Air }
    { platform:iOS Simulator, id:6032BE94-1A62-4ADB-9013-50FAB2B088F6, OS:8.1, name:iPad Air }
    { platform:iOS Simulator, id:D145A9C9-DFD0-4564-8A45-BF68A80426AC, OS:7.0, name:iPad Retina }
    { platform:iOS Simulator, id:F1967F38-BE03-430F-BFB1-AFF4E2E4511C, OS:7.1, name:iPad Retina }
    { platform:iOS Simulator, id:E8652514-3042-4958-9027-48A9A95392CE, OS:8.1, name:iPad Retina }
    { platform:iOS Simulator, id:6CF0F409-F4F8-4BCD-AC4E-30E58947A3EB, OS:7.0, name:iPhone 4s }
    { platform:iOS Simulator, id:57DBC56E-1E04-4384-9A07-7B47C119BEAB, OS:7.1, name:iPhone 4s }
    { platform:iOS Simulator, id:DDF255C3-8BEF-4A97-8B3F-AD651BABF6CB, OS:8.1, name:iPhone 4s }
    { platform:iOS Simulator, id:1C08AA1B-9C0C-499B-9570-972EF52941D9, OS:7.0, name:iPhone 5 }
    { platform:iOS Simulator, id:BAF484D4-803C-478D-BAC1-916376C22C48, OS:7.1, name:iPhone 5 }
    { platform:iOS Simulator, id:45E5CE55-093D-48C0-AA26-AA4E5C569FDB, OS:8.1, name:iPhone 5 }
    { platform:iOS Simulator, id:3DD4CAA5-6CD7-4DFB-AF95-6494EAE6E90F, OS:7.0, name:iPhone 5s }
    { platform:iOS Simulator, id:6856CCCF-842E-48E8-897B-87BB1865CD10, OS:7.1, name:iPhone 5s }
    { platform:iOS Simulator, id:0AC6BB28-E860-4320-A532-272BE43C067C, OS:8.1, name:iPhone 5s }
    { platform:iOS Simulator, id:E574C367-1BAA-4F9F-BB92-BD5F6BAB1226, OS:8.1, name:iPhone 6 Plus }
    { platform:iOS Simulator, id:7953EE9F-19E8-4D0C-9219-268FB0BA6626, OS:8.1, name:iPhone 6 }
    { platform:iOS Simulator, id:15FC1628-B369-44DE-8CE0-756439D9AEBC, OS:8.1, name:Resizable iPad }
    { platform:iOS Simulator, id:5E7A9E8C-D3A8-4E58-935F-8754B100E867, OS:8.1, name:Resizable iPhone }
    { platform:iOS Simulator, id:9151073B-3B7D-441A-8069-E797FA5059CE, OS:8.1, name:MyIPhone6 }
    { platform:iOS, id:5dc0e5a8a9f7c0af1feea4bfa13860c9e61350d6, name:Maksym's iPhone }

Thanks Tor Arne for pointing out -destination-timeout option in the comments. Without this option xcodebuild will take way too long to figure out that destination doesn’t exist.

Back to Destination

Now that last output from xcodebuild is much better than anything else. It gives you all the key-value pairs as is, no additional guesswork involved. You get the “platform”, “OS” and “name” keys, and as a bonus you get an undocumented “id” key. Give it a try and see that it actually works.

xcodebuild test -project MyProject.xcodeproj -scheme MyScheme -destination "id=E574C367-1BAA-4F9F-BB92-BD5F6BAB1226"

You can adopt this approach as part of CI workflow automation and grep all key-value pairs from xcodebuild output.

Install Java on Mac OS X

2015-02-15T00:00:00+00:00

A short how-to for installing and configuring Java Development Kit on Mac OS X.

Java is core technology used by CI servers like Jenkins, Bamboo and others. It is also used by Atlassian CLI Client. In fact installing and configuring JDK is such a common task that it deserves its own post.

Java Version

Select JDK version which you need: 1.6, 1.7 or 1.8. Better install all of them, you never know what you might need one day.

I find that for most applications I’m using 1.7, rarely 1.6 for some outdated and unsupported application. These days more and more applications are updated to work with 1.8 without issues.

Java Home

Lots of applications, if not all, expect JAVA_HOME environment variable to be set. This tells them where the Java Virtual Machine (JVM) is located. I would recommend to update your ~/.bash_profile and add something like this:

This is an over-defensive version. It can safely run on OS other than Mac, but will not set JAVA_HOME in that case. It uses OS X specific java_home executable found in /usr/libexec. A very convenient utility to get JVM path for Java installation. Try -V to get a list of all JVMs installed.

If you are using shell other than bash, include the same in your shell’s version of .bash_profile or source it directly with source .bash_profile.

CocoaPods for Enterprise

2015-02-15T00:00:00+00:00

A practical example of using CocoaPods for enterprise projects.

The “Enterprise” definition isn’t that clear and is not something standard. I will define it as follows.

Enterprise is a company that builds more that one app on the same platform.
Has a large number of in-house libraries and modules.
Has a rather complex project configuration.

The challenge is to adopt CocoaPods for such projects. This is actually more of a case-study than success story of adopting CocoaPods.

`xcconfig`

One thing that “complex” project setup means is extensive use of xcconfig files. The problem is that CocoaPods generates it’s own xcconfigs and expects them to be applied to project target. If it detects existing xcconfigs it displays a warning in the end of generation step. Solution is to follow the advise and include Pods xcconfig in the very last xcconfig file which is applied to a target.

// Pods
#include "Pods/Target Support Files/Pods/Pods.debug.xcconfig"

Make sure you do that for all targets and all configurations.

`$(inherited)`

This is next issue which is known to CocoaPods users.

The problem is that CocoaPods assumes it’s the only thing in the world using xcconfigs. So it never ever uses $(inherited) thus it doesn’t pick up any of the previous definitions when included into other xcconfigs.

One of the solutions is to patch Pods xcconfig files as part of pod update or pod install command. This is what post-install hooks are for. Consider this example

# Podfile for Project with xcconfig files
source 'https://github.com/CocoaPods/Specs.git'

platform :ios, '7.0'

link_with 'TargetName'
pod 'Facebook-iOS-SDK', '~> 3.20.0'

post_install do |installer|
    puts "Running post install hooks"
    puts "Patching Pods xcconfig files"
    workDir = Dir.pwd
    installer.project.targets.each do |target|
        target.build_configurations.each do |config|
            xcconfigFilename = "#{workDir}/Pods/Target Support Files/Pods/Pods.#{config.name.downcase}.xcconfig"
            xcconfig = File.read(xcconfigFilename)
            # insert $(inherited) for HEADER_SEARCH_PATHS and OTHER_LDFLAGS, make sure there's only one occurrence
            patchedXcconfig = xcconfig.gsub(/HEADER_SEARCH_PATHS = (?<first>[^\$])/, 'HEADER_SEARCH_PATHS = $(inherited) \k<first>').gsub(/OTHER_LDFLAGS = (?<first>[^\$])/, 'OTHER_LDFLAGS = $(inherited) \k<first>')
            File.open(xcconfigFilename, "w") { |file| file << patchedXcconfig }
        end
    end
end

The post_install hook iterates through all targets, figures out the full path to generated Pods xcconfig and patches it by prepending $(inherited) for all occurrences of HEADER_SEARCH_PATHS and OTHER_LDFLAGS.

Git Submodules and Proxy

This is another issue that may happen in “enterprise” environment, especially if proxy is involved. CocoaPods is not able to checkout pods with submodules. An example of a pod with such problems is Facebook SDK.

To avoid this problem in the first place I would recommend to use pods with static frameworks instead of those that clone whole source base. Well, not for all pods, but this is often useful for 3rd party SDKs that are not updated often. This is what pods like Flurry, Crashlytics and others do. For some reason Facebook doesn’t have official podspec for framework only, but nothing stops you from creating one yourself and host it in-house.

If you have to have those submodules, use the following git trick:

git config --global url.https://github.com/.insteadOf git://github.com/

See discussions on StackOverflow and this post. This command will modify your .gitconfig by adding this line

[url "https://"]       insteadOf = git://

This will solve submodule issue and is much easier than other workarounds like messing up with proxy stuff and tools like socat.

Git Submodules for Internal Components

Another blocker for migration is all those internal libraries you already use as submodules in your project. You still want to be able to change their code right inside submodule directory and work with git in-place. With pure CocoaPods approach this is not possible since all your changes will be reset on next pod update.

However, there’s a beautiful work-around that allows you to benefit from both worlds (submodules and pods). Here’s a great article on the matter.

The idea is to use so called Development Pods. That means in your Podfile you specify the path to a submodule directory which has podspec file. This way CocoaPods ignores version information in podspec and uses latest files from submodule directory to generate pods. You get all the benefits of working with submodules code directly and then get all the flexibility of CocoaPods approach.

Build Problems

Final paragraph is related to building projects with xcodebuild from command line. This is what happens on CI box after all.

If you try to run your old scripts which use custom configuration build directory you may run into link errors with Xcode not being able to find library file for Pods (libPods.a). Tools like xctool have this problem fixed, but I recall earlier version of xctool being prone to the same problem.

The solution is twofold.

If specifying custom CONFIGURATION_BUILD_DIR then make it an absolute path

# Bad.
CONFIGURATION_BUILD_DIR=build

# Good.
CONFIGURATION_BUILD_DIR=$(pwd)/build

If overriding CONFIGURATION_BUILD_DIR you must also specify OBJROOT, SYMROOT and DSTROOT and make sure all those are absolute paths as well

# Example.
OBJROOT=$(pwd)/build SYMROOT=$(pwd)/build DSTROOT=$(pwd)/build

# Example of complete build command.
xcodebuild clean build \
  -workspace Sandbox-ObjC.xcworkspace \
  -scheme Sandbox-ObjC \
  -configuration Release \
  CONFIGURATION_BUILD_DIR=$(pwd)/build \
  OBJROOT=$(pwd)/build \
  SYMROOT=$(pwd)/build \
  DSTROOT=$(pwd)/build

OCLint

2015-02-08T00:00:00+00:00

OCLint is a fantastic static code analysis tool for improving quality and reducing defects by inspecting C, C++ and Objective-C code.

I’m not going to copy-paste the rest of oclint.org here, check it out to see what the tool can do. In this article you will learn how to set it up and use for Mobile CI.

OCLint is often referred as a lint tool, but in the end of the day it’s a static analysis tool, meaning it analyses your code statically, that is without running it.

Install

For quite a while I though OCLint wasn’t available via Homebrew package manager. While it technically isn’t there yet, it’s supported by Homebrew Cask. If you never used it yet, Homebrew Cask is an extension for Homebrew and lets you to manage OS X applications just like packages. Another benefit of this tool is that it includes other packages like OCLint which are not yet available in Homebrew.

brew install caskroom/cask/brew-cask

Now install OCLint

brew cask install oclint

Check the installation, it’s now sitting in usr/local/.

Use

I will write about using OCLint with xcodebuild, basically rewording official documentation. OCLint supports all other tools, check out documentation for details.

Using OCLint with xcodebuild requires 3 steps.

Build and capture output.
Generate compile commands database (JSON format) for OCLint using xcodebuild output.
Run OCLint using generated JSON database.

The first step is straightforward, run xcodebuild the way you would run it normally, the only difference is that you want to capture build output. Use tee command to do that, for example

xcodebuild clean build -project MyProject -scheme MyScheme -configuration Debug | tee xcodebuild.log

It is recommended to use Debug configuration for static analysis.

Next, generate compilation commands database with oclint-xcodebuild command. This command expects a file named xcodebuild.log by default, but in the example I will pass it explicitly. The default output file name is compile_commands.json but I’ll be more verbose anyway and specify output file name.

oclint-xcodebuild -output compile_commands.json xcodebuild.log

Optionally you can use -e | -exclude option and pass regular expression which will exclude the files you are not interested in, such as 3rd party source code.

Finally, use oclint-json-compilation-database to run oclint and generate report.

oclint-json-compilation-database \
  -exclude ${LINT_EXCLUDES} \
  -- \
  -report-type ${LINT_REPORT_TYPE} \
  ${LINT_RULES} \
  ${LINT_DISABLE_RULES} \
  ${LINT_THRESHOLD} \
  -o ${LINT_REPORTS_DIR}/${LINT_REPORT_FILE} \
  -stats \
  -verbose \
  -list-enabled-rules

Let’s analyze this call in details and expand actual values of all environment variables.

First you call oclint-json-compilation-database and specify it’s own arguments. In this case it is a list of excludes passed as ${LINT_EXCLUDES}. OCLint understands grep-like regular expressions syntax, for example

LINT_EXCLUDES="Libraries|lib|Pods|Carthage"

The -i [INCLUDES] option is used to do the opposite to exclude.

The double dash -- indicates the start of arguments which will be passed to invocation of oclint command.

Report type (LINT_REPORT_TYPE) specified via -report-type can be one of text, html, xml, json and pmd. For CI tasks you should definitely use PMD report type. While running locally stick with html. So you should set LINT_REPORT_TYPE to one of these values. Read the full documentation for more.

Lint rules (LINT_RULES) is an option with which you can override default threshold values for rules. For example, Objective-C is very verbose language, so let’s increase thresholds for the the max length of line of code, name of method and name of variable. This is how it will look in shell script

# Rules.
LINT_LONG_LINE=300
LINT_LONG_VARIABLE_NAME=64
LINT_LONG_METHOD=150

LINT_RULES="-rc LONG_LINE=${LINT_LONG_LINE} \
      -rc LONG_VARIABLE_NAME=${LINT_LONG_VARIABLE_NAME} \
      -rc LONG_METHOD=${LINT_LONG_METHOD}"

You may want to disable some rules as well. This could be useful when working with some legacy code which smells a lot and there are no plans to fix it ever. In cases like this it’s easier to silent some warnings. For example, let’s disable warning about the use of _ivarName outside of accessors and initializers. Let’s also ignore useless parentheses and warnings about unused method parameters.

# Disable rules.
LINT_DISABLE_RULES="-disable-rule=UnusedMethodParameter \
            -disable-rule=UselessParentheses \
            -disable-rule=IvarAssignmentOutsideAccessorsOrInit"

Full list of rules for further reference.

Note!: The names of the rules on documentation page do not match the names you should be using to disable them. For example CoveredSwitchStatementsDontNeedDefault, should be specified as SwitchStatementsDon\'TNeedDefaultWhenFullyCovered when used in command line scripts (note escaping the ' too). To get the correct names use --list-enabled-rules option, which is mentioned further in the post.

Threshold allows you to control when analysis passes or fails. There are 3 priorities (Priority 1, 2 and 3) and each comes with default thresholds (0, 10 and 20). You can use -max-priority-<N> command line option to customize priority N threshold. This is how you would change it to 0, 20 and 30 correspondingly

# Threshold.
LINT_PRIORITY_1_THRESHOLD=0
LINT_PRIORITY_2_THRESHOLD=20
LINT_PRIORITY_3_THRESHOLD=30
LINT_THRESHOLD = "-max-priority-1=${LINT_PRIORITY_1_THRESHOLD} \
    -max-priority-2=${LINT_PRIORITY_2_THRESHOLD} \
    -max-priority-3=${LINT_PRIORITY_3_THRESHOLD}"

Output can be specified via -o option. If you are using PMD report type, write out to XML file, for HTML report, write to HTML format. Better create a separate directory for reports, for example

# Reports.
LINT_REPORTS_DIR=oclint-reports
# Use .html for HTML report type.
LINT_REPORT_FILE=oclint.xml

mkdir -p ${LINT_REPORTS_DIR}

Finally add some verbosity with -stats to output statistics, -list-enabled-rules to eyeball the rules being used and -verbose for what it stands.

Config File

Instead of having lengthy shell scripts, you can opt out to using .oclint config file. Put it in you project’s root folder and list all the thresholds and other rules configs it YAML format, for example:

# OCLint config example

rules:
disable-rules:
  - GotoStatement
  - UnusedMethodParameter
  - EmptyIfStatement
  - SwitchStatementsDon'TNeedDefaultWhenFullyCovered

rule-configurations:
  - key: LONG_LINE
    value: 300
  - key: SHORT_VARIABLE_NAME
    value: 1
  - key: LONG_VARIABLE_NAME
    value: 64

max-priority-1: 0
max-priority-2: 20
max-priority-3: 30

enable-clang-static-analyzer: false

Read official documentation for more details.

Other Notes

Integrate with Xcode

There are ways to integrate OCLint with Xcode. This is actually worth a separate post on its own. The goal is to allow non-script-and-command-line-savvy teammates to run OCLint analyzer in Xcode. Don’t set your hopes too hight though, they just won’t do that anyway :) However, check this post for some examples.

Suppress Warnings

Sometimes you just have to violate the rule. Let’s say it’s a legacy code base and there’s just this one violation and nothing you can do about it. Instead of disabling a useful rule completely, consider suppressing the warning.

Note: 0.10.1 doesn’t understand //!OCLint suppress comments. It is case-sensitive and must be //!OCLINT. This should be fixed in next releases though.

Save Time

Running a project build to generate Xcode build log, then running OCLint is very time consuming task. Consider generating Xcode build log once and then reusing it for OCLint. You will have to generate fresh build log when the project structure changes.

If you run unit tests as part of CI pipeline, then capture build log when building unit tests target.

Summary

Give it a go. You now have a very powerful tool in your tool belt. Figure out your custom project thresholds, integrate it as part of your project makefile or Fastfile, run it as CI job and output reports using one of the plugins available.

You may actually unearth a pair of real bugs in your code.

Makefiles for Mobile CI

2015-02-08T00:00:00+00:00

A look at Xcode projects from another angle, trying to build things on another level.

When I started with iOS development, not much really existed in my noob view outside the Xcode IDE. For quite a while Xcode used to be my whole universe. I could go through the whole app life cycle from first line of code all the way to uploading it to App Store, all of that in Xcode (except minor detour to iTunes Connect web interface). At some point my universe had to expand and I learned about basics of internal Xcode project structure, Xcode workspaces, xcconfig files, custom build phases and so on. Eventually I got out of the metaphorical Solar system (Xcode IDE) and dived into the black void of terminal window and command line interface.

Xcode Limits

In this article I will try to show you other ways to build Xcode projects. There’s nothing bad with Xcode projects as such. As I mentioned before, you can have quite an elaborate configuration with multiple targets, schemes, cross-target dependencies. You can include and build sub-projects, define custom build phases such as running shell scripts. Then finally you can use xcconfig files to take out project settings out of ASCII plist format and control them in plain text.

With all that goodness you will eventually get to the limits of what Xcode project can do. Here are some examples.

CocoaPods. You have to get out of your “comfort zone” and run pod install and pod setup from command line to get Xcode project all set up. Though you could create a custom target with a shell script pre-build phase to do just that.

If you have a lot of post-build phases with shell scripts, you probably want some of those to be executed only when you really need it. Have you ever seen hundreds of dSYMs uploaded to Crashlytics because that small shell script phase was executed each time a developer would hit Cmd + B? That’s just one example.

No matter how sophisticated is your use of xcconfigs, you’ll eventually want to build an app overriding some of build settings from command line. That’s often the case in CI setup, where you want to build with another provisioning profile, sign with another signing identity, etc. Going into the code and changing xcconfig manually doesn’t work for CI and it’s too manual. Having dozens of targets with xcconfigs that differ in one or two lines is another not so pleasant experience.

Back to shell scripts as custom build phases. Given that you have half a dozen of those, you will want to reuse them for other projects as well. Does copy-paste feel right? It should not and that’s yet another reason to see how building iOS projects can be taken to the next level.

Make Utility

The answer, well, one of the answers, is make utility and makefiles.

make is a build automation tool. It automatically builds programs and libraries from source code by reading files called makefiles which specify how to derive the target program. Phew, that’s enough of Wiki copy-paste. make was initially released in 1977! That’s years before I myself was “released” so to say. By ways of evolution and inheritance make is a part Mac OS X by default, so why not trying to use it for something good?

Makefile

make is using makefiles, which contain so called recipes often referred to as targets, a set of instructions for building the app, library, or whatever you are up to. Does make clean sound familiar now?

Makefile syntax is similar to shell syntax, but not exactly the same. The fact that you can use shell commands in makefile makes it a bit more confusing.

Basic Recipes

Let’s start with a set of simple targets. To define a target you normally tell what you want to build and after a colon (:) you tell how. Let’s create a file named Makefile.

PROJECT = MyProject.xcodeproj
SCHEME = MyScheme
BUILD_DIR = build

# phony targets
.PHONY: all clean help

# default target
all: help

# clean target
clean:
  @echo Cleaning up...
  xcodebuild clean -project $(PROJECT) \
    -scheme $(MYSCHEME)
  @rm -rf $(BUILD_DIR)

help:
  @echo Targets:
  @echo "clean - clean the project"
  @echo "help - display this message"

Let’s see what’s going on here. all is the default target which is executed when you just run make. Here you can see how target dependencies can be used. all is dependent on help and help is just printing information about all available targets. Other than causing help to run, all doesn’t do anything else.

Another trick - phony targets, those are all listed as dependencies for a special .PHONY target. Using phony targets is the way you can tell make utility to build those targets each time. By default make is smart and checks for any changes since last build and does nothing if it detects no changes. In this case I don’t want that behavior, that’s why I use phony targets.

The clean target cleans Xcode project using xcodebuild under the hood. It also removes the build folder. The actual recipe for clean is a number of shell commands. By default make will print out all the executed commands to stdout. The use of @ as in @echo will remove the “echo Cleaning up…” line form stdout and you will see only the “Cleaning up…” message. Another nice thing is that just like with shell you can use \ to split single command into multiple lines.

This example also features the use of variables in a makefile. You might have noticed that it’s less strict than the shell syntax, e.g. it allows use of spaces around assignment operation. Using $() you can reference variables, shell-like ${} is also a valid syntax. You could as well just use $ without () or {}, but I’d recommend sticking with $() all the time. Having $PROJECT resolved as $P ROJECT is not the easiest thing to figure out. Finally make lets you override variables from command line.

Try running make clean to see how it works.

make clean SCHEME=MyOtherScheme

make will look for a file named Makefile by default, though you can always feed it any makefile you want.

make -f MyOtherMakefile clean

Functions

At this moment it should be obvious that makefiles are using Domain Specific Language or DSL. It so happens that this language has functions among other things.

Let’s look at two targets, build and test.

PROJECT = MyProject.xcodeproj
SCHEME = MySceme

build:
  xcodebuild build \
    -project $(PROJECT) \
    -scheme $(SCHEME)

test:
  xcodebuild test \
    -project $(PROJECT) \
    -scheme $(SCHEME)

Looks a bit DRY doesn’t it? Here’s how you can define a function called xcodebuild to be able to reuse some code.

PROJECT = MyProject.xcodeproj
SCHEME = MySceme

# Build with xcodebuild
# usage: $(call xcodebuild,ACTION,PROJECT,SCHEME)
xcodebuild = \
  xcodebuild $(1) \
    -project $(2) \
    -scheme $(3)

build:
    $(call xcodebuild,build,$(PROJECT),$(SCHEME))

test:
    @$(call xcodebuild,test,$(PROJECT),$(SCHEME))

Well, to be honest, this doesn’t look like the best example in the world, but still it demonstrates the use of functions. You can pack some reusable code into the function, then call functions from functions and eventually gain benefit from this approach. Interestingly enough, when it comes to calling functions makefiles do not tolerate spaces, as you can see I put no spaces after , when passing arguments. Arguments are positional and not named, you have to refer to them with $(1), $(2) and so on. You can still use @ to filter out extra output from stdout.

To call a function you use $(call function-name,arg1,arg2,...) syntax.

Shell Commands

When writing code outside of target recipe, you can call shell commands using shell keyword.

PROJECT = $(shell ls -1 . | grep .xcodeproj)

This example will find first file with .xcodeproj extension and save it to PROJECT variable. If using CocoaPods you can modify shell script to filter out Pods.xcodeproj.

Here’s more advanced example, that demonstrates nesting of shell commands.

CLANG_ANALYZER = $(shell dirname $$(which scan-build))/bin/clang-check

Note how I have to escape $ with another $ when capturing results of which command, that’s because single $ is part of makefile syntax and will try to resolve $(which scan-biuild) as makefile variable but not as shell expression.

An example for shell for-loop.

SCOPE = -iregex ".*\.(h|m|mm)$$"
MAKEFILE_VAR = Value

for-loop:
  @(for file in $(shell find -E . $(SCOPE)); do \
    echo "Do something for $${file} using $(MAKEFILE_VAR)";	\
  done)

This code will loop through all files with .h, .m or .mm extensions. Again notice the use of double $$ for referring to shell variables vs single $ for makefile variables. The @(...) is used to wrap the whole expression and prevent it from showing up in stdout. It also demoes the use of multiline scripts. Here’s an example of a one-liner for-loop.

BUILD_DIR = build

one-liner-for-loop:
  for f in $(shell find . -name $(BUILD_DIR)); do echo Removing $${f} ...; rm -rf $${f}; done

In general, any shell script will work, just don’t forget to escape $s properly and keep an eye for makefile/shell syntax differences.

Flow Control

You have already seen that it’s possible to use all shell flow control operators in the recipe. If you want that level of control outside the recipe, you can use makefile’s own flow control operators.

For example, a basic if-block.

ANALYZER = xcode
CLANG_ANALYZER = Xcode
ifeq ($(CLANG_ANALYZER), scan-build)
CLANG_ANALYZER = $(shell dirname $$(which scan-build))/bin/clang-check
endif

Now by calling make with different parameter you can influence the value of CLANG_ANALYZER variable.

make analyze ANALYZER=scan-build
# Value of CLANG_ANALYZER will be /usr/local/bin/clang-check/

These flow control operators can be used outside as well as inside recipes. Have a look at foreach example.

PLATFORMS = linux mac windows
RUBY_FILES = $(shell find . -type f -name '*.rb')
RUBY_LOG = check-ruby-syntax.log

check-ruby-syntax:
  @$(foreach platform, $(PLATFORMS), \
      echo Platform: $(platform); \
      $(foreach file, $(RUBY_FILES), \
        export PLATFORM=$(platform); \
        ruby -wc $(file) 1>>$(RUBY_LOG) 2>>$(RUBY_LOG); \
      ) \
    )

This example is from a bit different world of Ruby. It iterates through a list of platforms (linux, mac and windows), then for each platform it iterates through all .rb files it can find. For each file it sets the shell environment variable PLATFORM and runs Ruby syntax check command (ruby -wc) writing errors and warnings to the log file.

One more makefile command worth mentioning is eval. It can be used inside the recipe to assign new value to makefile variable, for example

MAKEFILE_VAR = 0

eval:
  $(eval MAKEFILE_VAR := 1)

Of course you can use more complex expressions, e.g. assign a result of some function to MAKEFILE_VAR.

Includes and Reuse

If rich DSL syntax wasn’t enough, you can get another level of reuse from makefiles. You can include them to each other just like good old C header files.

Consider this example

# Actual directory of this Makefile, not where it is called form
SELF_DIR = $(dir $(lastword $(MAKEFILE_LIST)))

# Include common Batman Makefile
include $(SELF_DIR)/CommonMakefile

This is how you can include CommonMakefile into another makefile. The magical SELF_DIR construction is the way to get absolute path of the makefile itself. This is useful when you call make and give it a path to makefile which is located in some other folder and then CommonMakefile is located in the same directory with original makefile. Kind of require_relative if you ever dealt with Ruby.

Including makefiles suggests the ability to reuse recipes, and that’s possible indeed. To make recipe reusable use double colon :: when defining it.

# Nn CommonMakefile.
clean::
  echo "Common clean..."
  @rm -rf CommonBuildDir

# In SpecificMakefile.
clean::
  echo "Specific clean..."
  @rm -rf SpecificBuildDir

As you’d expect, the clean recipe in SpecificMakefile will first run clean recipe from CommonMakefile.

Configuration as a Code

Imagine that you’ve created a bunch of recipes in your makefile, for example

clean
build
test
analyze (e.g. Clang Static Analyzer)
lint (e.g. OCLint)
deploy (e.g. to HockeyApp)

You have practically created all the build tasks for your CI job. Depending on the size of your project, you can execute those separately, e.g. make test or combine into one recipe and run as one make ci command.

ci: clean build test analyze lint deploy
  echo "Running CI target ..."

All what is left to do is to collect reports. The beauty of this approach is that you don’t have to open Jenkins/Bamboo/Team City/Whatever UI to modify the build job each time. Your whole build configuration is a code now and you make changes inside the project repository, that means all changes are tracked in version control system and can be reviewed.

Even more, if multiple projects share the same structure, you can move build configuration into a common makefile and make it available on CI box as part of Ruby gem, for example. There are certain tradeoffs if you do it this way. The latest version of common makefile has to be backwards compatible with all the projects. I have successfully applied this approach building 21 libraries with one common makefile, so it’s something that can be done.

Summary

I’ll admit that this not the best use of makefiles. If you ever had to build complex C or C++ projects, you should be familiar with much more complex recipes and use of extensive pattern matching and other techniques. However, makefile is a DSL after all and the beauty of any DSL is that you can apply it to your needs in a way which is convenient for you.

Makefiles sometimes feel like something from last century, and they technically are! Nothing stops you from exploring other alternatives if you want to get away from this non-obvious makefile-shell syntax mixup and functions that look more like macros.

Have a look at Rake then. Basically it’s make for Ruby, but you can use it for any kind of projects. You get all the benefits of using Ruby as your DSL, which is way more flexible than makefiles. A minor tradeoff is that you need to have Ruby and Bundler installed, but Ruby is part of Mac OS X release, so that’s not a big deal.

Yet another option is Gradle. The DSL for Gradle is Groovy a powerful and flexible language as well. With Gradle you’ll need some basic Java setup.

If you want to take your iOS or Mac OS X build automation to the next level, just pick one of the many DSLs available and move all your configuration to the code. It does require certain efforts in the beginning, but pays off in the end.

Code Coverage for iOS (gcov)

2015-02-08T00:00:00+00:00

Create code coverage reports for iOS unit tests using gcov tool.

Note that this approach may not work with Xcode 7 and future Xcode updates. For information on how to generate test coverage reports using new Profdata format, check this article).

Calculating Code Coverage is a way to get more out of your unit tests, given that you have those and run on regular basis. With minor changes you can get coverage reports that include stats for

Packages
Files
Classes
Lines
Conditionals

Enable

Good staring point is documentation from Apple. Important takeaway from that article is that you need 2 sets of files to generate coverage reports. The .gcno files contain information to reconstruct the basic block graphs and assign source line numbers to blocks. The .gcda files are generated when the tests are executed and contain transition counts and some summary information. Check this link for more details.

The Apple’s article recommends to create a separate configuration and set the following LLVM 5.0 Code Generation options to YES.

Generate Debug Symbols (GCC_GENERATE_DEBUGGING_SYMBOLS)
Generate Test Coverage Files (GCC_GENERATE_TEST_COVERAGE_FILES)
- This is required to generate .gcno files.
Instrument Program Flow (GCC_INSTRUMENT_PROGRAM_FLOW_ARCS)
- This is required to get .gcda files.

I have also specified the GCC settings names so you’d know how to enable these flags when building from command line or when using xcconfigs. The benefit of building from command line is that you don’t have to create new configuration in the project, instead you can customize existing Debug configuration.

# Make sure tests are run against latest OS at all times.
export OS=$(xcrun --sdk iphonesimulator --show-sdk-platform-version)

xcodebuild test -project MyProject.xcodeproj -scheme MyScheme \
  -configuration Debug \
  -sdk iphonesimulator${OS} \
  -destination OS=${OS},name="iPad Retina" \
  GCC_GENERATE_DEBUGGING_SYMBOLS=YES \
  GCC_GENERATE_TEST_COVERAGE_FILES=YES \
  GCC_INSTRUMENT_PROGRAM_FLOW_ARCS=YES

If you’ve just ran this command you will be surprised to find no .gcda files anywhere. Apparently, writing these files to disk is a costly operation so this data needs to be flushed with explicit command. The way to do that is to call __gcov_flush in your app’s code. Apple recommends to do that when the app is sent to background, here’s the quote

Set the UIApplicationExitsOnSuspend key to YES in your Info.plist file and send your running application to the background by pressing the home button.

This is very un-automatic and very un-CI, Apple. This readme provides a list of other options. Well, the first one is reacting to the event of app entering background, which is not good. The second option uses method swizzling and swizzles tearDown method of XCTest class. This way GCOV data will be flushed when all the test cases are completed.

I am using swizzling as well, but swizzle tearDown method of XCTestCase. This way data is flushed every time a test case finishes.

#ifdef ENABLE_GCOV_FLUSH

@import XCTest;
@import ObjectiveC.runtime;

// If you have to build for versions below iOS 7.0, use this code instead
// #import <XCTest/XCTest.h>
// #import <objc/runtime.h>


extern void __gcov_flush();

@implementation XCTestCase (GCovFlush)

+ (void)load {
    Method original, swizzled;

    original = class_getClassMethod(self, @selector(tearDown));
    swizzled = class_getClassMethod(self, @selector(_swizzledTearDown));
    method_exchangeImplementations(original, swizzled);
}

+ (void)_swizzledTearDown {
    if (__gcov_flush) {
        __gcov_flush();
    }
    [self _swizzledTearDown];
}

@end

#endif

Note that I don’t have a header file. There’s no need for it since you are not going to include it anywhere. Another notable difference is the use of ifdef guard ENABLE_GCOV_FLUSH which needs to be defined when building from command line, this is just me being over paranoid about including any kind of non-production code in the app. Save this file and name it XCTestCase+GCovFlush.m Before you add it to your project, one very important note

The file must be added to the main app target, also called “Test Host”, not to the unit tests target. More details.

OK, so now you can add this file to your Xcode project. There are ways to automate this injection as well, e.g. using xcodeproj Ruby gem. I plan to have a write up about it some time later. Anyway, you can now run your tests and have .gcno and .gcda files at your disposal.

export OS=$(xcrun --sdk iphonesimulator --show-sdk-platform-version)

xcodebuild test -project MyProject.xcodeproj -scheme MyScheme \
  -configuration Debug \
  CONFIGURATION_BUILD_DIR=build \
  -sdk iphonesimulator${OS} \
  -destination "OS=${OS},name=iPad Retina" \
  GCC_GENERATE_DEBUGGING_SYMBOLS=YES \
  GCC_GENERATE_TEST_COVERAGE_FILES=YES \
  GCC_INSTRUMENT_PROGRAM_FLOW_ARCS=YES \
  GCC_PREPROCESSOR_DEFINITIONS="\$(GCC_PREPROCESSOR_DEFINITIONS) ENABLE_GCOV_FLUSH=1" \
  | tee test.log

Note that I’m redefining preprocessor definitions to enable ENABLE_GCOV_FLUSH and also want to reuse those already defined in Xcode project, that’s why I refer to them with help of escaping $ sign. I’m also capturing logs in test.log for further processing.

This paragraph has nothing to do with coverage reports, but you need to capture test results in a report format that most CI servers would understand. You can use ocunit2junit to convert OCUnit (now XCTest) report to JUnit format.

# install
[sudo] gem install ocunit2junit

# run
cat test.log | ocunit2junit

Default output is in test-reports directory, point your CI report plugin to this location to pick up XML and generate test reports.

Another option is xcpretty tool, which is yet another Ruby gem.

# install
[sudo] gem intall xcpretty

# generate JUnit XML reports (default location is build/reports directory)
cat test.log | xcpretty --report junit --color

# generate HTML reports (default location is build/reports directory)
cat test.log | xcpretty --report html --color

Note that you don’t have to re-run the tests. All that the tool needs is build log.

If you are OK with using something other than xcodebuild, check out Facebook’s xctool. It has few options to help you get test reports without the need of extra gems. It also supports parallel execution of tests, so definitely worth a look.

Report

It’s time to report the coverage results. gcovr is the right tool for the job. Since it’s a Python utility, here’s how you install it

sudo -E easy_install pip
pip install gcovr

Use -E option to tell easy_install to pick up current user’s environment variables such as HTTP_PROXY while running as super user. Don’t bother with this option if you can run easy_install as a non-sudoer, e.g. when you have custom Python installation.

Use -x or --xml to generate XML report. gcovr has issues with generating HTML reports, we’ll have to use another tool for the job later.

BUILD_DIR=build
GCOV_FILTER='.*/MyClasses.*'
GCOV_EXCLUDE='(.*./Developer/SDKs/.*)|(.*./Developer/Toolchains/.*)|(.*Tests\.m)|(.*Tests/.*)'
COVERAGE_REPORT=coverage.xml

gcovr \
    --filter=${GCOV_FILTER} \
    --exclude=${GCOV_EXCLUDE} \
    --object-directory=${BUILD_DIR} -x > ${COVERAGE_REPORT}

This example also demonstrates the use of filter and exclude options to filter unwanted files from reports. BUILD_DIR points to the directory you defined with help of CONFIGURATION_BUILD_DIR when running the tests.

The XML output is enough for CI tasks, but to have a sneak peek at readable HTML results on your computer you’ll end up nowhere with gcovr, the tool has a bug (unless it got fixed already). The tool to help at this time is lcov.

# Install.
brew install lcov

# Use.
BUILD_DIR=build
LCOV_INFO=lcov.info
LCOV_EXCLUDE="${LCOV_INFO} '*/Xcode.app/Contents/Developer/*' '*Tests.m' '$(BUILD_DIR)/*'"
LCOV_REPORTS_DIR=lcov-reports

lcov --capture --directory ${BUILD_DIR} --output-file ${LCOV_INFO}
lcov --remove ${LCOV_EXCLUDE} --output-file ${LCOV_INFO}
genhtml ${LCOV_INFO} --output-directory ${LCOV_REPORTS_DIR}

Here you can see another use of exclude option to filter out unwanted results. genhtml is bundled with lcov installation. More documentation on lcov can be found here. You now have browsable coverage report in lcov-reports directory.

Summary

The real point of calculating code coverage reports is to use them as part of CI process to keep track of overall project health and mark it as unstable or failed if coverage is less than required. Check out Jenkins vs Bamboo article for more details.

Clang Static Analyzer

2015-02-08T00:00:00+00:00

A brief post about Clang Static Analyzer and scan-build tool.

Clang Static Analyzer is a source code analysis tool that finds bugs in C, C++, and Objective-C programs. Yep, no Swift yet.

Xcode and `xcodebuild`

You may have used it already since a stable build of clang static analyzer comes bundled with Xcode. It’s ⌘⇧B (Command + Shift + B) shortcut in Xcode or analyze action when building from command line.

xcodebuild analyze \
    -project MyProject.xcodeproj \
    -scheme MyScheme \
    -sdk iphonesimulator \
    -destination "name=iPhone 6s Plus,OS=9.2" \
    -configuration Debug

Note the use of -sdk and -destination options. Latest Xcode versions may play dumb when used from command line and will want explicit destination specified. By default the build destination will be Generic iOS Device and will most likely require code signing.

When running Analyze action in Xcode you get a beautiful report with nice arrows rendered right on top of the source code. While running from command line the analyzer errors will show up in build log.

The build log by itself should be enough for integration with CI servers, but you can actually get more out of standard xcodebuild analyze action. Using CLANG_ANALYZER_OUTPUT and CLANG_ANALYZER_OUTPUT_DIR you can control in which form analyzer creates report and where. Valid options for CLANG_ANALYZER_OUTPUT are text, html and plist. You can combine multiple options using dashes, e.g. plist-html.

xcodebuild analyze \
    -project MyProject.xcodeproj \
    -scheme MyScheme \
    -sdk iphonesimulator \
    -destination "name=iPhone 6s Plus,OS=9.2" \
    -configuration Debug \
    CLANG_ANALYZER_OUTPUT=plist-html \
    CLANG_ANALYZER_OUTPUT_DIR="$(pwd)/clang"

There will be a lot of output files in clang directory, using find you can locate HTML report.

find clang -name "*.html"
# It will find a file named like report-f27e58.html

Open it in a browser and it looks just like the one in Xcode.

Under the Hood

If you are curious what are those not-so-much-documented CLANG_ANALYZER_ build settings, this section may shed some light for you.

CLANG_ANALYZER_OUTPUT=html translates into -Xclang -analyzer-output=html when xcodebuild composes and executes clang command. Similar story for CLANG_ANALYZER_OUTPUT_DIR. You can call clang -cc1 --help yourself, there are a lot of interesting things in the help message. For example, running clang -cc1 -analyze -analyzer-checker-help will list all the available checkers. If you look at build log closely now, you will see how xcodebuild configures all those checkers using -Xclang -analyzer-config and -Xclang -analyzer-checker flags.

With this knowledge, you should be able to tweak default Xcode Analyze configuration by modifying Xcode build settings. To be honest, I never had to do that. If you really want more control over static analyzer, you should look at scan-build tool.

scan-build

scan-build is a command line utility that enables a user to run the static analyzer over their codebase as part of performing a regular build (from the command line).

Why scan-build?

If Xcode comes with a static analyzer already available, why bother and use another tool?

scan-build has a newer version of static analyzer with a number of experimental checks enabled. For example, using Xcode 7.2.1 I got 0 analyzer warnings when running Analyze action on a given project. For the very same project scan-build finds 6 warnings related to nullability and detects possible nil values passed to methods that don’t expect nil.

Installation

Clang Static Analyzer is not available for installation via Homebrew directly. You can get it as part of llvm package if installed with --with-clang option, but this is not recommended, because Homebrew llvm version will not be the same as Apple LLVM version. You won’t be able to build your iOS projects with this toolchain right away.

So I would recommend to use the custom Homebrew tap instead.

brew tap mgrebenets/scan-build
brew install scan-build

Run scan-build to make sure installation is successful.

Usage

You can use it in two ways, from Xcode or from command line.

To learn how to use it from Xcode read this documentation. I never actually did it. Aware that others mention a number of caveats with this approach, some not so obvious configuration tweaks required to make it work properly.

I’m more interested in running it from command line and generating reports for CI tasks. Here’s the original documentation with some good examples.

The basic usage is supposed to be as simple as this

# Build a scheme
scan-build -k -v -v xcodebuild clean build \
    -project MyProject.xcodeproj \
    -scheme MyScheme \
    -destination "name=iPhone 6s Plus,OS=9.2" \
    -configuration Debug

Note the use of clean build and not just build. I have discovered that clean action may not be necessary and for me just running build action works as well and yields the same results.

By default scan-build will use static analyzer version bundled with its own installation (clang-check). You may choose to specify an explicit path to static analyzer executable or fall back to using the version bundled with Xcode.

# Use analyzer bundled with xcode.
scan-build -k -v -v --use-analyzer Xcode \
  xcodebuild clean build \
  -project MyProject.xcodeproj \
  -scheme MyScheme \
  -destination "name=iPhone 6s Plus,OS=9.2" \
  -configuration Debug

# Use explicit path to clang-check executable.
CLANG_CHECK=$(which clang-check)

scan-build -k -v -v --use-analyzer ${CLANG_CHECK} \
  xcodebuild clean build \
  -project MyProject.xcodeproj \
  -scheme MyScheme \
  -destination "name=iPhone 6s Plus,OS=9.2" \
  -configuration Debug

Reports

Use -o (output) option to specify output directory for reports.

mkdir -p clang-reports

# Build a target.
scan-build -k -v -v \
  --use-analyzer Xcode \
  -o clang-reports \
  xcodebuild clean build \
  -project MyProject.xcodeproj \
  -scheme MyScheme \
  -destination "name=iPhone 6s Plus,OS=9.2" \
  -configuration Debug

There are dozens of other options you can use to customize scan-build, try scan-build -h to see all of them.

It is also supposed to be easier to customize checkers with scan-build. I had never had a chance to do that, you must have some compelling reason to go beyond default configuration.

Troubleshooting

If you get an error like this: clang: error: unknown argument: '-fembed-bitcode-marker', make sure you have specified both -sdk and -destination options to xcodebuild.

Same applies for xcodebuild failing to find imports like #import <UIKit/UIKit.h>.

CI

I’ve already touched this topic in my Jenkins vs Bamboo comparison. For both CI servers your best option is to publish clang static analyzer reports as HTML page. Analyzer warnings and errors will also be picked up by Warnings plugin in case of Jenkins, and you are up to some grepping or other voodoo if you are dealing with Bamboo.

All in all clang static analyzer is another great tool to have at your disposal, especially if you are working on CI tasks.

References

Bitbucket Branches and Jenkins Job DSL

2015-02-08T00:00:00+00:00

A simple example of applying Jenkins Job DSL Plugin in real life.

For any real life application of any tool you have to start with real life problem or a goal you want to achieve. In this case, the goal is to generate Jenkins build projects for all Git branches of your project. Remote repository in this example is Bitbucket, which makes things a bit more different compared to dealing with GitHub.

You have probably ended up here by following the link from this article. To reiterate, there are couple of Jenkins plugins that come close to solving the original problem, but they have number of serious drawbacks. So let’s see how you can solve this problem with help of Jenkins Job DSL.

Get Bitbucket Branches

First, let’s start with example for GitHub.

The first 3 lines are hitting GitHub API and grab the list of all branches. This code is not going to work for Bitbucket, so we need to come up with something different. Another complication is that (in my case) we are dealing with private Bitbucket repository, so need to take authorization into account. So lets figure out what’s the URL to hit. The components of URL are

API Base URL - by default it’s https://bitbucket.org/api
API Version - 1.0 or 2.0
API Endpoint Path - includes the following
- “repositories” - since we want to use one of the repositories API
- Organization Name - aka team or account name
- Repository Name - repository slug
- Repositories API Endpoint - branches since we want to get list of branches

Time to put it all together

Next we need to convert this string to URL, hit it and parse the output. But before we do that, we have to set Authorization header for HTTPS authentication with username and password. The username and password should be Base64 encoded.

This code, when put together, will return list of all branches in JSON format, or will not in case you are behind the…

Proxy

This bit of code will help you to configure proxy for JVM

Yep, ";"s are a legacy thing and I put them there to demonstrate relation between Java and Groovy. In general, any Java code is a valid Groovy code, but not the other way around.

Filter Branches

The JSON returned by Bitbucket API is a dictionary. Each entry has branch name as a key and branch description as value. Branch description is yet another dictionary with entries such as author, last commit hash and message, timestamp for last update, etc. Here’s an example.

To experiment with Bitbucket API directly you can use this REST Browser.

Using this information you can filter out unwanted branches. The reason to do that is that not all developers do a proper cleanup after their branches are merged. You can end up with branches as old as 3 years or more, which you don’t want to pick up and create CI build project for. So you can use timestamp information to ignore branches, which haven’t been updated for a long time. This is also an opportunity to enforce correct branch naming rules and filter all branches with incorrect names.

An answer an absolutely valid question of “Why the branches are not deleted automatically on merge?” That’s because some versions of git-flow do not use Bitbucket’s native merge feature, but use a rebase instead. Thus the branches are left hanging around after they are merged and it becomes developer’s responsibility to delete the branch.

Let’s go through the code in case comments are not descriptive enough. The main part is iterating over JSON dictionary branchesJson returned by Bitbucket API. On each iteration we have branch name branchName and its details packed as details JSON dictionary. We get the last modified date timestamp and convert it into the Date object. Now we can check if the branch has valid name and is not too old.

Valid name in this example means that branch is one of the major branches (master, development and anything that starts with release), or that branch name has one of the 3 valid prefixes (feature, bugfix and hotfix). isValidBranch method splits the branch name by "/", gets the first element and checks if it’s one of the valid prefixes.

Another filter is for branches that are too old, or in other words branches that haven’t been updated for too long. This is what isUpToDateBranch method is for. Note that we consider all major branches to be always up to date. For example, master branch can be updated only when major releases occur and we don’t want its build project to be removed in the meantime. If Jenkins project is removed and then created again, its build number will be reset to 1, this is something we want to avoid, especially if build number is baked into the app version. Anyway, the logic is straightforward, if branch is one of the major branches, then consider it to be up to date. Otherwise, compare branch last modified date with current date and if the difference is more that expected (15 days in this example), then ignore this branch.

Then there’s a note, that no def keyword or type are used to declare majorBranches, validBranchPrefixes and allValidPrefixes variables. This is intentional. Omission of def makes these variables a so called binding variables. This is due to the fact that you are working with a Groovy script here and you have to declare variables like this to make them available for methods, e.g. to refer to them inside isValidBranch and isUpToDateBranch. I know it doesn’t sound like a solid explanation, but you have to take into account the fact that I myself should be considered as Groovy beginner, so this is the best I can come up with at the moment.

Final bit that needs some explanation, is the use of job construction. job is a property of the Groovy script you are running. In fact, the scrip itself is an instance of DslFactory class. Job is configured with a closure. The bare minimum it needs to create Jenkins Job (or as I refer to it in this article Project), is name, so I set it to current branch name replacing all "/"s with "-"s in the process.

Summary

As a summary for this post, you can try to run Job DSL script on real Jenkins. First or all, you need to have an access to Jenkins server. Next, you have to have Jenkins Job DSL Plugin installed. Another required plugin is Git Plugin.

OK, so I assume you’re looking at Jenkins dashboard right now. Go ahead and create New Item. Choose a name for your project and select Freestyle project type and click “OK”.

On new project configuration page you can leave everything “as is” for this example, the only thing you need to do is to add a new build step of “Process Job DSLs” type.

Now you can grab this DSL script gist and copy-paste it in Jenkins. Don’t forget to select “Use the provided DSL script” option first.

You should have noticed that DSL script you just copied doesn’t have the authentication and proxy bit enabled by default. I decided to make those steps optional so that you could run it in any environment. Since it points to a public repository, no authentication is required. Feel free to modify it to point to your private or public repo and to use your proxy.

OK, run it now and you should see something like this.

Here you have 2 Jenkins jobs generated for master and development branches.

This is just the beginning, from this moment on you can have numerous improvements. For example

Refactor one big monolith script into packages and classes, such as
- Class to work with Bitbucket API
- Class to configure network proxy
- etc.
Put the script into repository and modify DSL job to clone repo and run the script from it
And much more…

Xcode Derived Data

2015-02-01T00:00:00+00:00

What is Xcode Derived Data and why is it important?

“Clean derived data”. You might have heard this phrase each time you face some extremely strange build problems.

DerivedData is a folder located in ~/Library/Developer/Xcode/DerivedData by default. It’s the location where Xcode stores all kinds of intermediate build results, generated indexes, etc. DerivedData location can be configured in Xcode preferences (Locations tab).

So now and then you run into odd build problems. The more complex your project is the more chances you have to face them. Using Swift increases that probability a fair bit. DerivedData folder is also infamous for growing up to gargantuan sizes.

I have faced this issue when dealing with Jenkins CI server running as Launch Daemon. Special jenkins non-login user would run xcodebuild and that would create intermediate build files in DerivedData folder. But these new files would have access rights configured for non-login user. If you manually build the same project in the same folder while logged in as a “normal” login user you will mess up with files created previously by jenkins user. You will then see further build jobs failing while trying to write to DerivedData. Same applies to Bamboo or any other CI server running as Launch Daemon.

The lesson is - don’t mess up and run things manually in your CI server’s guts. Another lesson, don’t run CI Xcode build jobs as non-login user to avoid permissions problems. A practical advice to take home - clean Xcode Derived Data on regular basis on your CI box(es). You could create a cron job to do that, make it run some time after midnight and execute this simple shell command

rm -rf /Users/username/Library/Developer/Xcode/DerivedData/*

Note the use of full path in the command. That’s in case you do have multiple user accounts as part of your CI setup and the plan can run under any of those accounts.

If you happen to have multiple build agents running on multiple machines (either physical or virtual), then each of those agents should do it’s own cleanup.

Cleaning derived data might increase the time of first build for each project next day, but it’s a minor drawback. You will also claim free space back by killing DerivedData’s huge appetite.

For daily use on your development machine create a type alias in your bash profile.

typealias xcode-clean-derived="rm -rf /Users/i4niac/Library/Developer/Xcode/DerivedData/*"

Daemon vs Agent

2015-02-01T00:00:00+00:00

Comparison of Mac OS X Launch Daemon and Launch Agent in regards to Mobile CI.

Launch Daemons and Launch Agents are Mac OS X mechanisms used to launch and then control processes in the background. In regards to Mobile CI launch daemons and agents are often used to run CI server and agent instances.

Compare

Here’s the table with brief comparison.

	Daemon	Agent
Launch Time	System start	User login
User Type	Non-login	Login
Home Folder	No	Yes
Login Keychain	No	Yes
iOS Simulator	No	Yes
Provisioning Profiles	No	Yes

Let’s analyze each criteria in more detail.

Launch Time

Launch daemon starts at the time of system boot while agent starts at the time of user login. That usually means you would have to enable autologin option for the user account which is responsible to launching your CI server or agent.

User Type

To run at the time of system start launch daemon user has to be a non-login user. This means that the user can not login into OS X windows system, the only way to work with this user account is via SSH sessions. This will cause more problems when we look at next comparison criteria. Launch agent is a login user and that means she can login into windows system, install applications (if the user has enough rights) and do other things.

Home Folder

This is really a part of being login or non-login user. Login user as all the other standard users has a home folder in /Users. Non-login user has no home and if you need a place for it to store its files you will have to create one manually.

Keychain is important for Mobile CI, especially for iOS. Whether you build for Enterprise, App Store or AdHoc distribution you have to sign the app bundle. To be able to sign you need a private-public key pair in your keychain and you need to have access to that keychain. Public key is part of the certificate so when I reference public key I also mean certificate that contains it.

With login user you can install development credentials (private-public key pair) into Login keychain in a very simple way. With non-login user you have no keychain to start with. Then you have to keep it up to date and make sure you put all new keys in that keychain.

iOS Simulator

Last but not least is the ability to use iOS Simulator. Unit tests are essential part of any CI and you have to run them in iOS Simulator which is a GUI app, and that means only login user can run it. Have a look at this discussion with some comments quoting Apple who recommends to use launch agent.

Provisioning Profiles

iOS Provisioning Profiles are normally managed by Xcode and located in ~/Library/MobileDevice/Provisioning Profiles in the home folder of a login user who installed Xcode on the system. When you pass provisioning profile UUID to xcodebuild command it looks for the profile in that folder. So if you run xcodebuild as non-login user the lookup will fail and you will be forced to specify full path to provisioning profile.

One of the solutions is to create a symlink like this.

<non-login-user-home>/MobileDevice/Provisioning Profiles --> /Users/login-user/MobileDevice/Provisioning Profiles

I have witnessed situations where by accident or on purpose the permissions of symlink were changed, thus changing permissions and/or ownership of Provisioning Profile folder in login user’s home. Wrong permissions were causing problems to Xcode, it wasn’t able to sync profiles because of that.

Permissions Hell

Let’s assume you have CI server running as a launch daemon. As discussed above default local agents are not capable of running iOS unit tests. Essentially your whole Mac box is incapable of running a lion share of your CI jobs. The solution is to create a remote agent running on the same host and configure that agent to run as a launch agent.

Let’s further assume that you are dealing with some legacy setup and default non-login local agents actually run some jobs, they execute xcodebuild commands but never run unit tests. Now you add another build agent to run on the same box. Both these agents will share same Git cache, so don’t be surprised when eventually Git fails to checkout repository. I haven’t yet identified the core cause of this issue, but everything points to file permissions. Daemon and launch agent build nodes both run under different user accounts and create files with different permissions, so eventually they’ll run into the trouble when working with shared cache.

Summary

Let’s summarize all the benefits and drawback of using Launch Daemon.

Benefits:

Starts at system boot

Drawbacks:

No Keychain
Can’t run unit tests (iOS Simulator)
Can’t launch any GUI apps (e.g. Genymotion for Android)
Potential file permission issues when running alongside another build agents
Can’t easily access Provisioning Profiles

I would advise to avoid using Launch Daemons for Mobile CI servers or build agents at all costs.

If you really-really-really have to use Launch Daemon to fire up CI server, make sure it runs no Mobile CI jobs on it’s default local build agents, setup a remote build node instead running on the same box as a launch agent.

Jenkins Remote Node on Mac OS X

2015-02-01T00:00:00+00:00

A guide for setting up a remote build node for Jenkins CI server.

If you are up to a task to setup remote build node, that means that you already have a server or need to install it first.

A good place to start is Jenkins Wiki page with detailed description of all available options. Don’t forget to have a proper JVM installation first.

I would recommend to do first time agent installation via Jenkins UI Jenkins -> Manage Jenkins -> Manage Nodes. Select an option to create node, configure node’s properties (make sure root directory exists) and you are almost good to go. The preferred way to launch and agent is via SSH.

If Jenkins Wiki is such a good reference, then why bother with the post? - I bet that’s exactly what you are asking.

Well, again, the main focus is Mobile CI and part of any decent CI is tests, unit tests in particular. With iOS those must run in iOS Simulator which is a GUI application, which means you are facing Daemon vs Agent issue. Here’s a very good discussion on StackOverflow.

So remote node launched via SSH is running as Launch Daemon and has no context to run GUI applications. If SSH session is running under non-login user, xcodebuild test will fail with error code 139 or alike. Running SSH session under login user will not help either, at best the simulator will launch but tests will never start. That rules out an option of running remote node via SSH. Trying all the tricks like enabling development mode, updating user’s group and security settings, didn’t work for me, probably because that answer was actual for Xcode 5, but not 6 and above.

Next option would be launching slave agent via Java Web Start. Option with clicking button in the browser should be ignored right away, it hardly qualifies as automated approach. Using javaws it a bit better though requires you to login to slave agent via GUI and then run the command and answer one prompt by clicking Run button. Finally, you can run it in a headless mode

java -jar slave.jar \
  -jnlpUrl http://yourserver:8080/jenkins/computer/parasite/slave-agent.jnlp

Now that’s better, this will start a slave agent that is capable of running GUI applications and thus running iOS unit tests in simulator. If you are puzzled where does slave.jar come from, the answer is that it is put there by Jenkins server when you have created slave agent via Jenkins UI. The jnlpUrl tells an agent where the server is. The order in which agent and server are fired up seems to be important. Server first, agent second, otherwise agent may fail to start. I had this issue while using GUI mode launch via javaws and yet to confirm if it’s true for headless mode. Still you have to start an agent manually if remote node machine reboots.

I have one solution to offer. It is not the best I can think of, but it’s the easiest and reasonably reliable to start with. You need to turn the headless slave launch command into a Launch Agent. The Launch Agent will start at the time of user’s login, so make sure the machine is configured for automatic login.

Setting up Launch Agent should become a usual drill for you after you deal with Mac-based CI for a while. Create a plist file in ~/Library/LaunchAgents, name it what you like, for example org.jenkins-agent-a.plist and put the following content in it.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>StandardOutPath</key>
    <string>/Shared/Users/Jenkins-Agent-A/log/jenkins-agent-a-out.log</string>
    <key>StandardErrorPath</key>
    <string>/Shared/Users/Jenkins-Agent-A/log/jenkins-agent-a-err.log</string>
  <key>KeepAlive</key>
  <true/>
  <key>Label</key>
  <string>org.jenkins-agent-a</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/bin/java</string>
    <!--<string>-Djava.util.logging.config.file=/opt/hudson/slave/logging.properties</string>-->
    <string>-Duser.home=/Users/Shared/Jenkins-Agent-A</string>
    <string>-jar</string>
    <string>/Users/Shared/Jenkins-Agent-A/slave.jar</string>
    <string>-jnlpUrl</string>
    <string>http://yourserver:8080/jenkins/computer/parasite/slave-agent.jnlp</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
</dict>
</plist>

The /Shared/Users/Jenkins-Agent-A is your agent’s home. You should have create one when setting up agent via server UI and slave.jar should be already located in this folder. Now if agent machine reboots and autologin is configured, the agent will go back online automatically. If the server reboots though, I’m not quite sure what happens, my assumption is that agent will stay alive and will keep trying to reconnect to server and finally they both happily reunite. I’m yet to test how exactly it works in reality.

To manually start and stop an agent use following commands (typealias if you need to use them often)

# Start.
launchctl load ~/Library/LaunchAgents/org.jenkins-agent-a.plist

# Stop.
launchctl unload ~/Library/LaunchAgents/org.jenkins-agent-a.plist

# list (by label)
launchctl list org.jenkins-agent-a

One additional note. You can run a remote agent on localhost, which technically makes it not remote any more. This may be useful when for some reason Jenkins server is launched under non-login user session (Launch Daemon again) and you can’t use default master node to run unit tests.

That sums it up. To compare how things are in Bamboo universe check out this post.

Jenkins CI Server on Mac OS X

2015-02-01T00:00:00+00:00

A guide for setting up a Jenkins CI server on Mac OS X machine.

So you want to have Continuous Integration for Mobile in your company and your final choice of CI server is Jenkins. If your company is big and you are lucky enough the Dev Support or Dev Ops team will do all the heavy-lifting and install it for you. But if it’s not the case you might’ve just landed on a page that has something to help you out.

Install

A kind of warning first, avoid installing Jenkins as Launch Daemon. For detailed reasoning checkout out this article.

Jenkins Wiki offers a list of options for Jenkins installation but doesn’t mention Mac OS X. It mentions Docker though and I’ve heard nothing but good things about Docker. In this article I will stick with Homebrew.

You will need JDK to be installed and configured on your Mac before proceeding.

To install run a simple shell command.

Jenkins will be installed to usr/local and Homebrew will actually tell you right away how to turn it into a Launch Agent.

This recommends you to symlink Jenkins launch agent plist file to ~/Library/LaunchAgents but I would advise against it. As you will see next you will need to modify that file. That means if you ever upgrade Jenkins via Homebrew all your changes in plist will be lost. My recommendation is to copy it instead of making a symbolic link.

Even more, once installed via Homebrew I then delegate Jenkins upgrades to Jenkins itself. For this reason I pin Homebrew formula to prevent Homebrew from upgrading Jenkins files.

Now you also have manual control over Jenkins installation and can start/stop it from command line.

Configure

To understand why you need to change plist try to run Jenkins server. Give it a go, create a couple of build projects that do some basics like checking out git repository and running simple build command. Very soon you should get an error message saying that Jenkins has ran out of memory. This seems to be a common issue with JVM and Mac OS X, Bamboo installations run into same problems. I’m not quite sure why default configuration doesn’t account for this, probably this is Mac specific and other operating systems are OK. Anyway, you need to modify default plist file for Launch Agent. Here’s what you need and might want to change.

JVM Virtual Memory and Garbage Collection

Tell JVM to use a 64-bit data model if available (-d64).
Set minimum and maximum heap size with -Xms and Xmx flags. 512 Mb works for me most of the time.
Configure garbage collector, class unloading and permanent space.

HTTP Proxy

By far the largest source of issues and frustration, company proxy. Specify it using -D option.

Port and Prefix

Run Jenkins on a custom port with custom prefix in url. This example uses default 8080 port and /jenkins prefix, so you can access your Jenkins dashboard like http://yourhostname:8080/jenkins or ever http://youthostname/jenkins. These arguments need to be passed to jenkins.war which was installed by Homebrew to /usr/local/opt/jenkins/libexec.

Run at Load

Enable Run at Load option to start server automatically if machine reboots.

Environment Variables

If any of the commands in this plist need environment variables this is how you can define them.

Standard Output and Error

It is up to you to redirect stdout and stderr. While sounds like a good idea for logging I would advise against redirecting stderr into a file. I once had to deal with 90 Gb log file created by Bamboo remote agent over a few months period.

Note that Jenkins put its files in .jenkins folder in your user’s home path. You also have to specify full paths when dealing with launch agent plists. Create log folder if it’s not there yet.

Other

By default Jenkins enables security protocol for email. I have also faced an issue with Bitbucket Plugin and had to set preferIPv4Stack flag as a workaround. These are all flags for java command.

Full Config

Now put it all together.

Now you have a reliable Jenkins server that runs 24/7 and performs stable CI tasks.

Tips

To find out how exactly Jenkins was launched, grep active processes list.

The output will tell you everything you need to know.

Other Ways

Jenkins Runner

There are other ways to install and start Jenkins server, one of them is using jenkins.sh runner script. It is not bundled with Homebrew installation by default, you should download it manually as mentioned on the Jenkins Wiki page.

In this case all the configuration options are in data/wrapper.conf file, you can check default file and easily figure out where to add your custom options.

The runner shell script itself can be launched as Launch Agent or Launch Daemon. Overall this is just a higher level of configuration.

Legacy Runner

Another approach I’ve seen is to use custom runner script. I am actually working with one right now but I suspect this is a legacy version of jenkins.sh.

The main difference is that all configuration is stored in Mac OS X defaults and then read by the script like this.

Defaults are stored as plist and are read as a dictionary. An example output looks like this

Using [sudo] defaults write you can change Jenkins configuration.

Obviously this is less preferred way than using wrapper.conf. Using OS X defaults leads to configuration which is non-reusable on other operating systems and can’t be easily put in SCM if needed.

Summary

A short summary - install with Homebrew, configure as Launch Agent. To configure Jenkins for Mobile CI tasks you can read other articles in this blog.

The configuration is far from being final. You will have to install plugins, configure SSH keys for git repositories and perform multitude of other administrative tasks to bring your Jenkins CI box up to speed.

Bamboo Remote Agent on Mac OS X

2015-02-01T00:00:00+00:00

A short guide for installing and configuring Bamboo Remote Agent on Mac OS X.

The starting point for an agent is a server, you should either have it configured for you or install yourself.

Atlassian documentation is always a good place to start. Next important thing is to have Java installed on remote agent machine.

Configure Server

You start by configuring server. Open Bamboo server dashboard in the browser and navigate to Bamboo Administration page (via Settings button in top right). Surely you have to be admin to get there.

Now select Agents on the left and click “Enable remote agent support”.

You can click “Install remote agent” now.

Install Agent

You are now looking at the page with further instructions. Just like Jenkins build agent Bamboo Remote Agent is a Java JAR file that needs to be downloaded to a target agent machine and then ran there.

Before you proceed you need to configure remote agent machine. Just like Bamboo server works well with a special user, remote agent will benefit from running under a designated user account. Create a Standard account from System Preferences, in this example we will name it bambooagent so this user’s home will be /Users/bambooagent. To make the whole process easier from now on I am assuming that you are currently logged in as bamboo-agent user.

Just like Jenkins remote agent, Bamboo remote agent needs an installation and home directories. So let’s create these two.

cd /Users/bambooagent
# Create installation directory.
mkdir bamboo-agent
# Create home directory.
mkdir bamboo-agent-home

Installation directory is where you need to put agent JAR file and then this JAR will download other required files and keep them in installation directory. Updates will occur in this directory as well.

Home directory is the location for all the build jobs metadata, intermediate build files, artifacts and so on.

Now it’s time to install, first get the JAR and put it into installation directory.

cd /Users/bambooagent/bamboo-agent
wget http://your-server-host:8085/agentServer/agentInstaller/atlassian-bamboo-agent-installer-5.7.2.jar

Or simply manually download it and put in the directory.

It is time to run the JAR and let Bamboo complete installation process.

java -Dbamboo.home=/Users/bambooagent/bamboo-agent-home \
  -jar atlassian-bamboo-agent-installer-5.7.2.jar http://your-server-host:8085/agentServer/

bamboo-agent-home is the default name for Bamboo home directory, but it doesn’t hurt to be more explicit and specify it.

If you have to take proxy into account, add more -D options, for example

-Dhttp.proxyHost=mycompany.proxy.host
-Dhttp.proxyPort=8080
-Dbamboo.agent.ignoreServerCertName=true

The last one is useful when dealing with self-signed certificates.

Fix the Broker URLs

As usual, nothing just works right away, as if it was designed that way…

For example, if you just experiment on your development machine where Bamboo server is already up and you want to run agent on the very same hardware, you would naturally use localhost as host name.

# Example for localhost.
java -jar atlassian-bamboo-agent-installer-5.7.2.jar http://localhost:8085/agentServer/

# Output log.
INFO   | jvm 1    | 2015/02/19 08:51:49 | 2015-02-19 08:51:49,438 WARN [Thread-0] [BambooActiveMQConnectionFactory] Broker URI: tcp://10.5.50.2:54663?wireFormat.maxInactivityDuration=300000 is invalid: java.net.ConnectException: Network is unreachable

OK, so it doesn’t work. Bamboo agent tries to hit so called Broker URI, which is pointing to the server, in this example it’s “tcp://10.5.50.2:54663?wireFormat.maxInactivityDuration=300000”.

Note the highlighted parts. First the fact that Broker URI uses an IP address (10.5.50.2 in this example). Second and more important is the port number in Broker URI (54663). Basically this means “say hello to Firewall!” and I’ll get back to that later.

If you change localhost to 127.0.0.1 the results will be the same, that’s because Broker URI is part of Bamboo server configuration and has to be set manually.

This is where I have to add a note that I’m trying this on the train with no internet connection, so basically my laptop is offline. I also have HTTP_PROXY environment variable set, so no wonder agent can’t resolve Broker URI with an IP address (even after I unset HTTP_PROXY variable).

Anyway, let’s figure out what is that Broker URI and where it can be changed. For that open Bamboo server dashboard, navigate to Administration page and find “General configuration” menu on the left in “SYSTEM” category.

Here it is, the question remains how to set it correctly. Here are the rules I learned the hard way.

Local Server

This is the case when both the server and agent are on the same local area network. Have a look at this discussion and this one too.

Ultimately in LAN configuration you have to use your server’s hostname with .local suffix. To get the hostname run hostname on the server machine.

hostname

# Example output.
R5003398.local

Remote Server

In this setup your server is remote from the agent’s point of view. For example your company uses Bamboo OnDemand and hosts server and elastic agents in EC2 cloud. But Mac agent is not an option for the cloud, so you end up hosting it in-house, meaning it’s sitting behind company firewall.

Another thing learned the hard way is to configure Broker client URL using IP address and not the host name. Don’t ask me why, I don’t have an exact answer. Probably this is specific for Mac agents. Most likely I missed something important, but here I give you the solution that works. This means the Bamboo server IP address needs to be static, which isn’t usually a problem and is preferred configuration anyway.

Apply Changes

You can change Broker URI using Bamboo UI, but you can also configure it with text editor. Bamboo server keeps it’s configuration in home folder (/Users/bamboo/bamboo-home if you followed this guide). This is how properties look like.

<property name="bamboo.artifacts.directory">${bambooHome}/artifacts</property>
<property name="bamboo.config.directory">${bambooHome}/xml-data/configuration</property>
<property name="bamboo.jms.broker.client.uri">failover:(tcp://R5003398.local:54663?wireFormat.maxInactivityDuration=300000)?initialReconnectDelay=15000&amp;maxReconnectAttempts=10</property>
<property name="bamboo.jms.broker.uri">tcp://0.0.0.0:54663?wireFormat.maxInactivityDuration=300000</property>

If you are making changes via UI you will see a message saying that you need to restart all remote agents. Guess what, that’s not true. You can give it a go and restart an agent and observe no changes in failover URI. The truth is that you have to restart the server for changes to take effect.

Try It Out

So in my example I change the Broker client URL to use local hostname.

failover:(tcp://R5003398.local:54663?wireFormat.maxInactivityDuration=300000)?initialReconnectDelay=15000&maxReconnectAttempts=10

Now I start an agent and this time it works.

INFO   | jvm 1    | 2015/02/19 10:56:57 | 2015-02-19 10:56:57,794 INFO [Thread-0] [RemoteAgent] * Bamboo agent '172.28.82.121' ready to receive builds.
INFO   | jvm 1    | 2015/02/19 10:56:57 | 2015-02-19 10:56:57,794 INFO [Thread-0] [RemoteAgent] * Remote Agent Home: /Users/bambooagent/bamboo-agent-home
INFO   | jvm 1    | 2015/02/19 10:56:57 | 2015-02-19 10:56:57,794 INFO [Thread-0] [RemoteAgent] * Broker URL: failover:(tcp://R5003398.local:54663?wireFormat.maxInactivityDuration=300000)?initialReconnectDelay=15000&maxReconnectAttempts=10

If I navigate to Bamboo server Agents page, I can see remote agent being online and ready to pick up builds.

Launch Agent

Next step is to make sure that remote agent starts automatically in case Mac box restarts. Start by enabling autologin for babmooagent user, this way system will login into the user’s account and fire up all of its launch agents.

To make the best use of Bamboo remote agent, it should be started as a launch agent, see Launch Daemon vs Agent for more details.

Create com.atlassian.bamboo-agent.plist in /Users/bambooagent/Library/LaunchAgents. This is an example file.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
   <key>Label</key>
   <string>com.atlassian.bamboo-agent</string>
   <key>ProgramArguments</key>
   <array>
   <string>/Users/bambooagent/bamboo-agent-home/bin/bamboo-agent.sh</string>
   <string>console</string>
   </array>
   <key>RunAtLoad</key>
   <true/>
   <key>ServiceDescription</key>
   <string>Atlassian Bamboo Mac OS X Build Agent</string>
   <key>StandardErrorPath</key>
   <string>/Users/bambooagent/bamboo-agent-home/logs/bamboo-agent.err</string>
   <!--
   <key>StandardOutPath</key>
   <string>/Users/bambooagent/bamboo-agent-home/logs/bamboo-agent.out</string>
   -->
   <key>KeepAlive</key>
   <true/>
   <key>UserName</key>
   <string>bambooagent</string>
</dict>
</plist>

As you can see it’s all standard stuff. Specify a label (Label), run agent at load (RunAtLoad), keep it alive and run the session as bambooagent user. There’s a detailed description provided and error output redirected into a file. You can see that I’m reluctant to save standard output to a file, that’s because the size of this file can become really big over time. Note that this is standard error and output for the agent runner, it may not necessary contain actual agent logs.

Finally, to start the agent a bamboo-agent.sh runner script is used from Bamboo agent home directory.

A usual practice is to have type aliases in your favorite shell to start and stop the agent from command line if needed. In this example I’m using launch agent label to control it. The same can be done by using launchctl [load|unload] plist-file.

alias start-bamboo-agent="launchctl start com.atlassian.bamboo-agent"
alias stop-bamboo-agent="launchctl stop com.atlassian.bamboo-agent"
alias restart-bamboo-agent="stop-bamboo-agent & start-bamboo-agent"

You may have noticed that there are no arguments for proxy, Java home and other settings. That’s because all these settings can be set in /Users/bambooagent/bamboo-agent-home/conf/wrapper.conf.

# The Bamboo Agent home configuration file
wrapper.java.additional.1=-Dbamboo.home=/Users/bambooagent/bamboo-agent-home
wrapper.java.additional.2=-Dbamboo.agent.ignoreServerCertName=false
#wrapper.java.additional.3=-agentlib:yjpagent

# Application parameters.  Add parameters as needed starting from 1
wrapper.app.parameter.1=com.atlassian.bamboo.agent.bootstrap.AgentBootstrap
wrapper.app.parameter.2=http://localhost:8085/agentServer/

In this example you can see how Bamboo agent home directory is set, as well as other settings including server URL (in this case it runs on localhost). To add proxy put more additional Java arguments

# The Bamboo Agent home configuration file
wrapper.java.additional.1=-Dbamboo.home=/Users/bambooagent/bamboo-agent-home
wrapper.java.additional.2=-Dbamboo.agent.ignoreServerCertName=false
wrapper.java.additional.3=-Dhttp.proxyHost=mycompany.proxy.com.au
wrapper.java.additional.4=-Dhttp.proxyPort=8080
# and so on...

Using wrapper.conf is preferred over jamming these arguments in launch agent plist.

Firewall

If you agent sits in corporate network and server is in the outer world (e.g. EC2 cloud), you will face some nasty firewall issues. Have a look at official Atlassian documentation. Ports number 8085 and 54663 (unless you specified different numbers) are used for bi-directional communication between agent and the server. Best practice is to open these ports in both directions. While lower port numbers are usually open, ports like 54663 need an exclusion rules configured in your company firewall. From experience this task can be either hard or impossible. Did you ever have that feeling that Network team is an extra layer or red tape and are there not to help you but to do the opposite? Welcome to the club then :).

To answer your question why Mac agent cannot be in the Cloud as well, see bits of my rant in Bamboo vs Jenkins and this discussion.

Summary

From this moment on you need to configure your remote agent’s Mac to be able to run Xcode builds and tests, as well as other CI tasks.

This is one of the ways to configure and run Bamboo remote agent.

If you are into Docker, check out this link. I’m only planning to give a try to this apparently awesome tool.

Bamboo Group Agent

2015-02-01T00:00:00+00:00

Bamboo Group Agent plugin for Mac agents.

Bamboo Group Agent is a free plugin for Atlassian’s Bamboo CI server. Like most plugins Bamboo Group Agent plugin is designed to solve a particular problem.

Problem

Consider a distributed CI setup, for example Bamboo Cloud. The server runs in EC2 Amazon together with a number of Elastic Agents. Your goal is to add Mac remote build agents to this CI setup.

First of all, there’s no way to have a virtual Mac agent in the cloud, not with Amazon at least. You either have to choose one of the Mac colocation providers or have a self-hosted Mac box for an agent. This is only half of the problem though. The other half is the way CI is usually configured.

Due to the nature of CI server and elastic agents setup, all elastic agents are usually identical clones of each other. With a few clicks of a mouse you can go from 1 to 25 agents, all of them initializing from the same image and having same specs from the start. As part of agents initialization, tools and software packages are installed as well, e.g. Git, RVM, npm, and so on. Those are normally called capabilities in Bamboo terms. The purpose of capabilities is to have more granular control over build jobs. The job specifies capability it needs to run, agents declare capabilities they have, and then Bamboo matches jobs to agents. The real issue is that such a powerful mechanism is usually not used. Since all the agents are identical, developers create their build plans with that in mind and thus they specify no capabilities in their build plans, because all the agents are capable of running their build jobs. This means the jobs can start on any agent.

This setup works until the moment you need to have a special agent with special capabilities, e.g. Mac agent with xcodebuild capability. You will usually have just one of those and it will be a very valuable resource used to build all the iOS projects in your team or even company. As you might have guessed, as soon as you add Mac agent to the agents pool all the other plans will start jumping on this agent and bear in mind that there are dozens if not hundreds of those plans. That’s the drawback of ignoring capabilities.

The problem is identified, so what’s with the solution? Going though the agents and declaring capabilities will not help, because all the plans need to specify capabilities as well.

Fix The Plans

Fixing the plans by specifying capabilities is the right solution. Unfortunately it is often a hard solution. Depending on your company you may have dozens if not hundreds of build plans, created by different developers in different periods of time. It’s a hell of a task just to track responsible people if you ever find anyone. Then you’d have to explain why it is important to specify capabilities and so on. It may take months before changes are applied and you don’t have months, you need Mac agents up and running right now.

This leads you to using a workaround which is…

Bamboo Group Agent

Allow me some copy-paste here.

Marking an agent as a “Group Agent” will prevent any Build Plans from using it, regardless of capabilities. Plans must request agent specifically.

Let’s see how this applies to our Mac agent. Start by installing Group Agent plugin from Marketplace. You will have to restart Bamboo server to apply changes.

Configure Group Agent

Next you need to configure the agent. In this example I will use remote agent running on the same machine. Navigate to Bamboo Administration, then select Agents and select remote agent from the list.

Then click “Add capability”.

Select “Group Agent” capability from drop-down list. Ignore “Allow Deployments” option for now, I will give more details on it later. Now you can click “Add”.

To verify results, find Group Agent capability in the list and click “Edit” (this option will give you a bit more info than just “View”).

Everything looks fine except for one thing, the agent name. Right now it’s the IP address, which makes it a bit tedious to remember and use in settings. It’s a good time to change this now. Click on the agent name link, then click “Edit details” button.

Use Group Agent

Now create or select a build plan you want to assign to the Group Agent. Then select Actions > Configure plan.

Next select “Miscellaneous” tab and tick “This plan requires a Group Agent” checkbox.

Note, that you are using agent name here and you can specify more than one agent, by providing a comma separated list of agent names, for example “Mac Agent 1, Mac Agent 2, Mac Agent 3”.

That’s it, this plan will now run only on Mac Agent 1 and no other plans will run on this agent or any group agents you configure in the future.

Drawbacks

The benefit of using Bamboo Group Agent is that it solves original problem described in this article. However, this approach has a number of drawbacks you need to know about.

Lock Entire Plan

First drawback is that by using Group Agent you lock the entire plan to be executed on certain group of agents only. Entire means the plan and all of its jobs will run on a single agent at a time, that means jobs can’t run in parallel and you can’t take advantage of this awesome feature. Another issue is that your typical iOS build plan will have only few jobs that actually need Mac OS X (xcodebuild), the rest will be tasks to upload builds for distribution, to generate reports, to update issue trackers and so on. Ideally all those tasks could be ran on other agents thus freeing up Mac agent for next xcodebuild job, but that’s something that can’t be done with Group Agent.

The reason for this is not plugin developer’s fault. Actual reason is the lack of flexibility of APIs provided by Atlassian in their plugin development SDK. Group Agent uses BuildAgentRequirementFilter class which allows to filter list of Bamboo agents capable of running the plan. There’s no API to apply similar filter for Build Jobs at the moment.

Deployment Projects

Another drawback is related to Deployment Projects. By enabling Group Agent for remote or local agent you exclude it from the pool of agents that can run Deployment Projects. I bet you’ve just recalled that “Allow Deployments” checkbox I have recommended to leave unchecked. Well this option does exactly what it says, but it opens up the group agent for all deployment projects. Once again you will have dozens of unwanted deployment projects starting on a group agent, most of those will fail since Mac agent does not necessarily have all the required capabilities to deploy.

Why would you need a Mac deployment agent? - you’d ask. Well, there’s quite a few deployment tasks that can be done on Mac OS X only. One example - uploading app binaries to iTunes Connect. For that you can use iTMSTransporter or an amazing deliver utility.

One of the solutions for this problem is to have a dedicated deployment agent. This is part of Bamboo’s “out of the box” functionality. While configuring deployment projects you can assign agents to run those deployment projects. Deployment agents automatically become excluded from build agents pool though. You can use this solution if you have a spare Mac box to do just that. Given the iTunes Connect deployments are usually rare (unless you upload to new TestFlight very often), this may result in a valuable resource being idle most of the time.

Another solution would be to support group agent feature for deployment projects. Since deployment project is always linked to a build plan, it would be nice for deployment project to inherit group agent settings from it’s “parent” plan. As a matter of fact, there’s an already merged pull request addressing this issue, though the changes didn’t make it into release yet.

Summary

If you have a good opportunity then do your best to promote proper use of capabilities in your CI setup. Otherwise give Bamboo Group Agent a shot.

Bamboo CI Server on Mac OS X

2015-02-01T00:00:00+00:00

This article covers the details of installing a Bamboo CI Server on Mac OS X.

Install

The preferred way to install Bamboo is using official Mac OS X installer. Starting point for documentation is official install guide. Follow installer steps using default values.

This article assumes you are dealing with Bamboo version 5.2 and above. All examples are for version 5.7.2.

To download tar-ball using command line run this command

Bamboo needs Java to be installed, recommended versions are 1.6 or 1.7, I would advise to go with 1.7.

Make a new home for Bamboo. It could be /Applications/Bamboo or ~/bamboo/bamboo-home or whatever you want. For this post I’ll stick with second option. It is often recommended to create a special bamboo user and put all things in their home directory. To create a new user use System Preferences and create Standard user.

Note that I switch user (su) to work with Bamboo user’s files and folders. This way I stay away from possible permissions conflicts. You could login as bamboo user instead. In any case, the rest of the article assumes you are operating as bamboo user.

Once downloaded unzip the tar-ball and copy atlassian-bamboo-5.7.2 to bamboo user’s home folder. Rename it to just bamboo to make it version agnostic.

Then go inside that folder. This is referred as installation directory and this is where you will install and run Bamboo from. Edit the file in atlassian-bamboo/WEB-INF/classes/bamboo-init.properties. Uncomment the line with bamboo.home and put the following.

This is where Bamboo will put all the customizations and build plan details.

Now you should run bin/start-bamboo.sh.

Bamboo is now running on http://localhost:8085/.

Configure

Next step is to configure Bamboo, heres official documentation. Recommended way is to use Setup Wizard which presents itself when you open http://localhost:8085/ in the browser.

Start by getting a license. You can always get a 30 days free evaluation license from Atlassian.

Next select Custom Installation option. Have a look at Base URL, make sure the IP address is correct. Also this is a good moment to pause and double check that your future Bamboo server has a static IP address allocated. This will become really important when dealing with remote agents. Leave the rest unchanged and click Continue.

Select Embedded database option. It is recommended to use other DB setup for production system. I might refine the article later to include setup details for PostgreSQL database.

Next, choose to create a new Bamboo home, configure your username. Give it some time and you will be able to create your first build plan.

Launch Agent

Now it’s time to make sure Bamboo start automatically when build box restarts. As mentioned in this article recommended way is to use Launch Agent. Start by enabling automatic login for bamboo user in System Preferences.

Next create com.atlassian.bamboo.plist in /Users/bamboo/Library/LaunchAgents.

Make sure you create few aliases to start and stop Bamboo server from command line.

Troubleshooting

This section provides an example of issues you may run into while trying to fire up Bamboo.

For example, you start Bamboo and see this instead of dashboard.

To find out the reason look into Bamboo logs, specifically in /Users/bamboo/bamboo/logs/catalina.out.

OK, so database is presumably locked by another process. In my case it meant there was another Bamboo running already. Double check it then by running ps.

Indeed it is. I have another terminal tab open running bamboo user session and Bamboo server.

After I stop or kill all the instances I run it again and this time it works.

This is just one example of troubles, all of them will pretty much present themselves via the same starting error page. Another common case is database upgrade, you can run into this issue if using PostgreSQL and upgrade it via Homebrew, so it’s recommended to pin PostgreSQL instead.

Summary

This will get you going with basic Bamboo CI server. There’s a number of additional administrative tasks yet to be done. Like configuring user access, integration with other Atlassian products, network configuration, plugins, capabilities and much more. I wouldn’t envy anyone who would have to deal with all those tasks. Companies that take CI seriously normally have Dev Support team keeping whole infrastructure running, not just Bamboo.

Bamboo vs Jenkins

2015-01-29T00:00:00+00:00

A biased and subjective (and by now a bit outdated) comparison of Bamboo and Jenkins as CI servers for mobile development, based on practical experience with both.

Continuous Integration and Continuous Deployment (Delivery, Distribution) has been around for quite a while. But surprisingly enough on a global scale it pretty much just got into its teen years in regards to mobile development. Well, subjectively, of course.

You can see all levels of mobile CI these days. Some would still install builds from Xcode, others would have a quickly patched up build server under their desk. Xcode Bots meet the needs of yet another group of people. Travis CI is good and for open source projects it’s probably the best option. And guess what, I know few successful iOS development companies that develop apps for enterprise clients, and have no CI at all.

The advanced level of CI would include distributed build systems with multiple build nodes, support for automated unit and UI tests, running tests on simulator and physical devices, automatic deployment to TestFlight, Hockey App, Over the Air, and much more. It becomes not just mobile development, but spans into areas like DevOps and others. Etsy’s blog post is somewhat outdated but still a very good example of where this path can take you.

If you decide to take mobile CI under your total control, you have to pick a build server to start with.

I personally have worked with Bamboo for 1.5 years and I’m dealing with Jenkins right now, so I have few insights and can give some comparison of the two.

Setup & Configuration

Bamboo installation and Jenkins installation tasks are about the same in terms of time and knowledge required. While installing one of the two you’ll climb a certain learning curve which will help you heaps if you ever have to deal with second option.

Both are built using Java. Being Java applications, both will require similar JVM configuration. Default configuration won’t really serve you well. You’ll experience out of memory issues as soon as you add a couple of basic build plans/projects.

And lot of other things are similar: configuration behind proxy, login vs non-login user (Launch Agent vs Daemon), OS X Keychain, iOS Simulator, etc.

GUI

Obviously, this is not a comparison criteria at all. This criteria is as subjective as it could possibly be!

Out of the box Bamboo UI looks much better than Jenkins version.

With Jenkins there are ways to improve your day to day user experience. You can customize theme and even make your custom UI improvements, like adding “Build Now” button where you like it.

You should start by installing Simple Theme Plugin and then configure it with one of the available themes. Not all the themes will look good, it all depends on the Jenkins version you have and browser you use, etc. I tried a bunch of them and ironically enough the only theme that looks good on our production CI box is called “Atlassian”. But I’m dealing with slightly outdated Jenkins version, you could get better results with up to date Jenkins.

There’s multitude of UI, List View and Page Decorator plugins available for Jenkins. Some of them must be good and help you customize you Jenkins look & feel and functionality.

Plugins

This is where the large community plays its role. Subjectively or not, Jenkins has a larger choice of plugins of all kinds, starting from management and organization of build jobs and ending up with reporting.

Via a fancy pie chart diagram we are told that Bamboo has 151 plugins on Atlassian Marketplace.

A quick check on Jenkins Wiki and we get a count of 1,071 plugins.

Reporting is one of the most important plugin categories in my opinion. For example, I prefer to have full control over Xcode build tasks and use makefiles combined with shell commands wrapping xcodebuild rather than use Xcode plugin. Part of the reason is to be able to migrate to another build server with less effort and to be able to run CI tasks on my development laptop. Same goes for other things like uploading to / downloading from S3 bucket, transferring files with rsync or scp, and so on. But in no way I can come up with scripts that will produce good looking reports including HTML formatting, diagrams and images, plus integration into Bamboo or Jenkins UI. For this task I prefer to use plugins.

Jenkins features 127 plugins just for reporting, that’s almost as much as Bamboo can offer in total. Of course, quality of all those plugins deserves a detailed research. Just numbers don’t tell much. But this post is based on some hands on experience, so I’ll compare some reporting plugins I have used over time.

Publish HTML

If you ever used Clang Static Analyzer for your iOS projects, you know then that there’s no proper reporting plugin for this task. You end up with HTML report that needs to be published and made available in build project/plan summary page.

With Bamboo you create a new Shared Artifact, with Jenkins you use HTML Publisher plugin.

Unit Tests

Of course you run unit tests as part of your CI, don’t you? In that case you are well covered by reporting plugin on both Bamboo and Jenkins. Jenkins plugin also includes trending graphs, which is nice.

Cobertura Code Coverage

Of course you don’t just run unit tests, but also gather code coverage data, don’t you?

This is the first time Bamboo falls one plugin short. You can check related discussion for more details.

Jenkins has your back on this one with Cobertura Plugin. You can configure custom threshold values to mark builds as failed if you tests don’t provide enough coverage.

Static Analyzers Reports

OCLint is amazing tool to run another round of static analysis on your code and detect vast number of issues as well as to enforce coding guidelines. OCLint can produce output compatible with PMD output format. So it can then be processed by Jenkins PMD Plugin. You end up with browsable report with issues grouped by priority, category and other criteria. You can navigate all the way down to the line of code causing the warning. Once again, you can configure threshold values to mark builds with lots of warnings as failed.

In fact, the same reporting plugin should be able to pick up output of Clang Static Analyzer which I mentioned before. However I couldn’t figure out the way to make Clang Static Analyzer to spit out correct XML file.

With Bamboo, unfortunately, all you have is publishing HTML report via shared artifact.

P.S. I’m a big fan of Swift programming language, but one thing makes me a bit sad. It will take some time before we get tools like OCLint available for Swift.

Warnings

If you run CI tasks such as build, test, analyze and lint, often you don’t want your build project/plan to stop immediately if it encounters an error, for example test error. You still want your reporting tasks to run and pick up those errors and generate reports for them. This often leads to a problem where you have compilation error but the build is marked as passing. Preferred way to avoid this issue is to have a reporting task which will pick up all the errors and mark build as failed if needed.

Jenkins has Warnings plugin for that. It will scan build logs and detect warnings and errors generated by compiler, and it includes support for clang so xcodebuild is well covered. All you need is to configure thresholds and fail builds if it has compile errors.

I don’t know about Bamboo plugin to do the same job. I remember myself few months ago grepping test logs for errors and then marking builds as failed by doing something like exit 1.

Functional Tests

If you happen to use frameworks like Calabash that produce Cucumber test reports, then Both CI servers have plugins to provide nice reports.

Higher-Order Plugins

Forgive me this injection of so popular these days functional slang, but this is how I want to introduce The Mother and The Father of all Jenkins plugins - Jenkins Job DSL Plugin. In short, it allows you to manage your configuration as a code and to generate and update build projects from code. What could be better for a developer?

Check out these resources to find out more.

I plan to write a post about my personal experience with Job DSL plugin.

Jenkins Job DSL plugin was a game changer for me personally. When I started working at new place a while ago, I had year and a half experience with Bamboo and acted as a strong Bamboo advocate initially. Until the moment I discovered unlimited power of Job DSL plugin. I personally can’t imagine going back to Bamboo and dealing with UI to update dozens of similar plans by hand.

That said, I still like Bamboo very much, particularly the way it integrates with the rest of Atlassian products. I hope Atlassian will implement Build Plan DSL plugin or provide API to make it happen. Bamboo Trade Depot Plugin is the only thing that could match Job DSL plugin, but unfortunately it’s not even close.

Build Plan/Project Structure

There is a bit of confusion caused by terminology used by Bamboo and Jenkins.

Bamboo

With Bamboo you start by creating Build Plan.

Each plan consists of one ore more Build Stages. Stages run in sequential order. If one stage fails next stages are never executed. Stages can be configured as manual to be triggered by hand.

Stages contain Build Jobs. Build jobs in one stage can run in parallel if there’s enough build agents for that. Each job can require different capabilities and can be dispatched to run on some designated build agent. Build jobs may produce Build Artifacts and share them with other jobs in consequent stages. Since jobs are parallelized, if one of them fails other jobs will still keep running until they finish on their own. This feature can significantly reduce build time, instead of sequential Build ⟼ Test ⟼ Analyze ⟼ Lint running 10 minutes, you get parallel Build || Test || Analyze || Lint running for about 3 minutes (given that longest job takes around 3 minutes).

Finally each job is made of Build Tasks. Tasks run in order from top to bottom. A task may be a basic shell script or one of the many tasks provided via plugins. Here’s a generalized example of a job and it’s tasks. One large and important group of tasks is reporting tasks.

Checkout git repository
Build
Test
Deploy
Generate test report

A summary of a Build Plan structure

.
└── Build Stage
│   ├── Build Job
│   │   ├── Task
│   │   ├── Task
│   │   └── [more tasks]
│   └── [more jobs]
└── [more stages]

Not So Parallel

Bamboo deserves a special side-note in regards to parallel job execution and iOS build plans.

As I mentioned before, the way you assign build jobs to local or remote agents is via capabilities. An agent defines capabilities it has, a job declares capabilities it wants, and then Bamboo matches the two.

If you are in total control of your CI setup and mobile team is the only one using your particular Bamboo server and all agents, you have all the power to set all the agents’ capabilities and then enforce a requirement that all build jobs created by you or your teammates must explicitly specify which capabilities they need. This way you’ll harness the full power of distributed configuration, all build jobs will run only on the agents they really should run on.

Another situation is when mobile CI is a new addition to company CI setup. There is already a CI server and few dozens of build agents and a lot of other teams using this CI setup. Lots of teams with lots of plans created over the years.

Now imagine that you are adding your specialized Mac build agent to be used for Xcode builds only. You setup and configure remote Mac agent, connect it to the server and… all the other build plans start jumping on your Mac agent! That’s because 99% of those plans declare no capabilities they require, they simply expect that all the agents are identical in terms of capabilities. And that works because all the agents are indeed identical clones. Well were identical clones until a new Mac agent was added.

There’s no easy fix to this problem and you can tackle it in one of 2 ways.

Ideal Solution

Ideally, CI must be done right. All the build plans must be maintained, updated and removed if no longer needed. As a requirement, all build plans must explicitly declare capabilities they require to be able to run. This is something to be enforced at team management level. Company has to have guidelines for creating and managing build plans and there must be a person or even a team (Dev Support team) responsible for keeping guidelines up to date and enforcing them.

Down to Earth Solution

The reality is rough. The number of existing plans is overwhelming, it will take months to chase people responsible for each build plan and communicate the importance of declaring capabilities to them. The whole change has to be made in a safe way so it doesn’t break existing workflows and production deployments.

You don’t have months of waiting allocated in your schedule, you need mobile CI running ASAP. One thing you could do is to setup your own CI server just for mobile and essentially move to “under the desk” setup. This way you get no support from Dev Support team (given that you have one) and all the trouble of setting up, configuring and supporting the server and agents is now yours.

However, you still can do mobile CI as part of company wide CI. There is a plugin that will help you - Bamboo Group Agent plugin. Have a read if you interested, Group Agent plugin offers a solution which is not fully flexible, but will help with original problem.

Jenkins

With Jenkins you start by creating Build Project, which is on occasions called Build Job causing certain confusion. In this post I’ll stick with Project term.

By default all you get is a basic Freestyle project that includes

Description
Parameters
Build Triggers
Build Environment
Build Steps
Post-build Actions

If you draw an analogy to Bamboo, then all you get is a build plan with single stage containing single job and list of tasks (Build Steps and Post-build Actions are nothing but tasks). That’s it. There are no stages and no way to run anything in parallel.

This is where plugins get into play. Mutlijob Plugin does exactly the same thing as Bamboo. Stages are called Phases, phases include Jobs. Jobs run in parallel when possible, while phases execution order is sequential.

One very important distinction is that jobs in multi job project are actually references to existing build projects.

.
└── Multi Job Build Project
    └── Build Phase
        ├── Build Job 1 ⇢ Build Project 1
        └── Build Job 2 ⇢ Build Project 2

In theory you can have a multi job project that includes a job which is a multi job project… You could even unintentionally create build job retain cycles and an infinite build loop.

To satisfy your craving for code Multijob Plugin support is added to Job DSL Plugin.

Once again with a fair bit of work Jenkins can match Bamboo’s default functionality and then with another fair bit of work can surpass it.

I mentioned artifact sharing briefly for Bamboo. For Multijob plugin there is no proper artifact sharing support yet. There’s a number of open tickets (JENKINS-20241, JENKINS-25111, JENKINS-16847, JENKINS-16847) with workarounds available. So the problem can be solved but it’s not part of official Jenkins release yet.

Not So Parallel Either

Jenkins is still prone to the same problem Bamboo is when it comes to adding iOS build nodes to existing infrastructure. Chances are high your Mac build node will be used by all the other build projects if capabilities are not configured properly.

Actually, in Jenkins world there are no capabilities as such, instead labels are used. Semantics are a bit different, but they serve the same purpose after all. Each build node can be labeled with multiple labels and build jobs can use flexible logical expressions to specify target nodes they want to run on. But then, if labels were not used in your corporate CI from day one, the amount of work required for labelling existing build projects can be too big.

At this moment I am not aware of a Jenkins analogue of Bamboo Group Agent.

Branch Management

Branches are an essential part of any source code management workflow. By supporting branches in CI workflow you get a number of benefits.

Timely Feedback

By running CI for each branch you provide developer with earlier feedback on their changes. They will know right away if changes they made are breaking the build, unit or functional tests, introduce static analyzer or linter warnings, etc. Developers can then fix these problems before they create pull request and involve the rest of the team in review process.

Earlier Tests

If you have testable build for each branch, you can make it available to your test team automatically as part of continuous delivery process. This way important and critical changes can be tested before they are merged to upstream branch (doesn’t mean you don’t do integration testing after the merge though). Bugs can be caught earlier and you end up with faster develop-and-test iterations.

Finally, some branches are never destined to be merged upstream. They may contain some experimental feature code, something you still need to build and make available to testers or internal stakeholders. For example have a special build to demo crazy design idea to your UI/UX people.

Branch Naming

I had to mention branch naming in a separate paragraph. For some reason, yet unknown to me, when it comes to naming a branch developers get completely wild. What I mean is that within the same team you can see branch names with and without prefix, using dashes or camel case, dashes and camel case combined, underscores… just anything. Sometimes the same developer would be very inconsistent when naming their branches. The very same developers are very disciplined when it comes to coding, i.e. class, variable and method names, and following coding style guide in general.

The upshot is that you have to have branch naming guidelines in your team, e.g. as a part of Git (other SCM) workflow. If the whole “issue” doesn’t look like an issue to you, wait until you have to manage this branchy havoc in regards to CI.

Bamboo & Branches

Bamboo does a great job with branches. With a single tick of a checkbox you can create branches of a build plan. By the way, you did understand it correctly, you create a branch of a build plan. Essentially, Bamboo clones the original build plan and changes its source repository configuration to point to a different branch. These plans still share same build phases, jobs and tasks, but you can configure some of the branch plan settings differently, that includes notifications, branch merging strategies, etc.

With a standard Java regular expression you can filter branches by their name and instruct Bamboo to ignore branch names that don’t follow guidelines. For example, the regex below will accept only master and development branches, and branches that are prefixed with bugfix/, hotfix/ or feature/ followed by JIRA issue ID and lowercase-with-dashes description.

A good Java regex test website is RegexPlanet.

This is a perfect way to indirectly enforce correct branch naming. After missing couple of builds developers will figure out what’s wrong and change their habits. In certain situations you can always add a branch manually via Bamboo UI or CLI tools.

With another simple configuration field you can control aging of the branches. If branch hasn’t been updated for a long time, Bamboo will remove its build plan. Of course you’d want to preserve certain branches forever and there’s yet another checkbox to do that.

I’ve mentioned branch merging strategies and that’s one more feature that Bamboo provides out of the box. Using Branch Updater or Gatekeeper you can configure build plan to do upstream merge before or downstream merge after running the build, and even push merged changes back to the repository. This is a very good way to detect merge conflicts earlier and to keep your git workflow in order.

Bamboo’s branches support also shines when it comes to CI pipelines, more on that later in the post.

Jenkins & Branches

Surprisingly, plugins initially do more harm than good in this case.

On fresh setup Jenkins has no Git support (SCM I get to work with and thus choosing it as an example). You need to install plugin to work with Git and you will most likely install the most popular Git plugin. This plugin is very powerful but of all the things it does not create branches for build projects. Well to be honest it shouldn’t.

When you look for a solution you will think of trying some other plugins first. Thre are a few: Multi-Branch Project, Feature Branch Notifier and others. The common problem with all of them is that they have their own support for SCMs including Git, that means you don’t get to use the Git Plugin and all of its powerful features, instead you end up with a very limited version of it.

I tried Multi-Branch Project plugin, it was OK, but not good enough. I couldn’t get branch filtering to work, it kept picking up all the branches ignoring filters. There’s no option to configure branch aging, no branch merging strategies and so on. Finally, no simple and easy support for branched CI pipelines.

This is what I meant by harm in this case. You spend time trying different plugins and none of them would match Bamboo default functionality.

But all is not lost. Once again you will be saved by Job DSL plugin. Paired with Git Plugin it can work miracles. While Git Plugin can’t branch build projects it can work with branches as such. With one line of Groovy code you can fetch all the branches for your repository including information on their last update. Then with simple string regex match and date comparison you can filter branches just like Bamboo does, and then you can generate build project for each branch, organize them into folders and views to match Bamboo’s UI. Code a bit to generate branch pipelines, same for merging strategies and much more.

Follow this link to find an example of getting branches for GitHub repository.

If you are using Stash there is no direct support for it via Jenkins plugins afaik. But as long as you have API access you can follow Bitbucket example to have basic support for working with branches. If you don’t feel like coding too much, you can wrap some shell scripts in thin layer of Groovy code to get same results.

Yes, it takes time and learning to get used to Job DSL plugin, but the benefits are huge.

Pipelines

Pipelines is another name for CI/CD workflows. In certain cases single build plan/project is not enough, and that’s when you can create pipelines. Creating a pipeline means you chain build plans together, if first plan (aka parent) is successful it triggers its child plans. Child plans can have children of their own, and so on.

.
└── Parent Plan
    ├── Child Plan 1
    │   ├── Grandchild Plan 1.1
    │   └── Grandchild Plan 1.2
    ├── Child Plan 2
    └── [more children]

A real world example of pipelines for mobile space is UI automation tests, often called Functional or Acceptance tests. UI automation is usually a heavyweight task compared to unit tests. If you run UI automation tests as part of a single build plan it can take too long to complete.

You can create a separate plan for UI automation only, but then you don’t want to trigger it every time there is a commit to SCM. There’s no point running UI automation tests if the build fails. So you add UI automation tests plan as a child plan to the main build plan thus creating pipeline.

This is just one example.

Bamboo Pipelines

Bamboo has support for pipelines out of the box. In parent plan configuration you simply add child plans and configure the way those are triggered, e.g. only when parent is successful.

Child plans don’t have to be configured in any special way, they are completely unaware of being some other plan’s children by default. The very same plan can be included in multiple pipelines acting in a different role.

The real power of Bamboo pipelines is in its support for branches. Child plan will only be triggered if it has the same branch as the parent plan. This means you have to configure branching for both plans to make it work. Normally this also means that both plans are using same repository, but it is not a mandatory requirement. If 2 different repositories have the same branch the feature will work all the same.

When child plan starts it can get artifacts from the parent via Artifact Downloader task. Yet again, the branch of the child plan gets artifacts from the matching branch of the parent plan, all is handled by Bamboo.

Jenkins Pipelines

Jenkins has no pipelines support by default. As usual plugins should be used.

This is something I’m only starting to work with, so this paragraph doesn’t have lots of details.

In general, Jenkins plugins let you match Bamboo functionality with a certain amount of work.

Most popular plugins used for pipelines are

Both are supported by Job DSL allowing you to script complex pipelines in code and change them easily.

Distributed Builds

Both Bamboo and Jenkins have support for distributed builds. Bamboo is using Remote Agents while Jenkins calls them Remote Nodes, sometimes referred as slave nodes or agents.

A side note - both servers support local build agents/nodes as well. Those are running locally (as the name suggests) on the same hardware as the build server.

Back to remote agents/nodes. Complexity of setting them up and configuring doesn’t vary much between the two CI servers.

Both will suffer from issues caused by sitting behind the company proxy, specifically if the server is located somewhere outside your company network (e.g. in AWS cloud).

Why the server is in the cloud and the agent/node is not? - you’d ask. Well, that takes us to next topic.

Mac Support

Right, this whole comparison is focusing on mobile CI after all, meaning you have to deal with one of the most popular mobile platforms these days - that is iOS.

To build an iOS app you must have Xcode, which can run only on OS X (unless you want to follow the path of certain insanity and make it work on other OS).

Hackintosh

Not a very good idea I’d say. The company does iOS development and wants to go with Hackintosh to setup OS X build server. Really?

Cloud (aka OnDemand)

Could be a really good option, but not with industry giants like AWS. AWS or alike is where you’d go if using Bamboo OnDemand feature. I can find posts as old as 2011 discussing this issue: one, two. Apple doesn’t make it easy to run virtual OS X instances to the extent that none of the big cloud providers are brave enough to provide official support for this feature. These days you can go with one of the many Mac Mini colocation services. You either rent a Mac hardware or even provide your own.

Self-Hosted

This is also an option. If your company has security concerns about those Mac hosting providers, or doesn’t want to spend money for that, or for any other reason, you can always purchase Mac hardware and host it in your data center.

Whenever you are using self-hosting or remote hosting, you end up dealing with native hardware. I find that Mac Minis with maximized CPU, RAM, and SSD storage are perfect candidates for iOS CI box. The more you have the better.

Next step is to install remote agent/node and get it running. As I mentioned already, installing remote agent for Bamboo is around the same complexity as installing remote node for Jenkins. The problems start popping up when you begin using them.

Bamboo Remote Agent

Things you definitely need to know in regards to Bamboo remote agent.

[The rumor has it that] Atlassian are using Mac OS X to develop their products, including Bamboo, but OS X is never ever listed as officially supported. Indeed, why would you choose to run your JIRA, Stash, Bamboo, whatever, on OS X server? Hopefully with increasing demand for iOS CI Atlassian will put a bit higher priority on fixing Bamboo issues for OS X, and there’s plenty, btw.

Before you even start using remote agent on OS X, you have to experience a bit of pain when trying to set it up.

Major and the most important group of issues is related to Artifacts Sharing.

Whenever your remote agent finishes a build stage it is most often producing build artifacts. It doesn’t matter if those artifacts are shared or not, they must be copied to the server anyway. That is part of distributed build system. Your remote agent can go offline any time, your build plan jobs can run on different agents with different capabilities and artifacts must be passed from one agent to another. All in all, the reality is - the artifacts must be copied to the server.

Problem Proxy

Is your remote agent behind proxy? May I introduce you to HTTP 1.1 Chunked Transfer Encoding then? Well, not to the feature itself, but how it relates to Bamboo: BAM-5182, more.

Bamboo server requires support of HTTP chunked transfer feature of 1.1 protocol version to pick up artifacts. If your proxy doesn’t support this feature, you are in trouble. Strictly speaking, this is not Bamboo’s problem, this is the problem of your company network team. HTTP 1.1 standard was released in 1999! There is a lot of HTTP proxy implementations that support it, nginx for example. However, things move really slow in most of big companies when it comes to changing network infrastructure. If you are so unlucky and you company is still stuck around 1999 in terms of network infrastructure, you will most likely have to find a work-around rather than waiting months and months before you get any progress on proxy upgrade.

Problem Atlassian

But wait then! Too early to take all the blame off the Atlassian’s shoulders!

First of all, Bamboo remote agent JAR totally disrespects JVM flags for HTTP proxy whitelist (nonProxyHosts), upvote! You can find ways around this issue, for example re-routing network calls using tools like SquidMan, but then you will face The Final Blocker: BAM!, BAM!.

Yes, Bamboo remote agent is 27 (twenty seven!) times slower than plain scp (secure copy over ssh) command when it comes to copying a single .zip file which is around few hundred Mb in size.

Imagine that after spending all the time trying to figure out issues around your proxy and enable the agent to share artifacts with the server, you end up facing this beast? It renders all your distributed build setup useless, it takes way more time to copy build artifacts than to produce them. From reports it looks like this issue is specific for Mac OS build agents only.

When I faced this problem I ended up sharing artifacts via Amazon S3 bucket. This is extra work, extra shell scripts to upload and download artifacts, additional expenses for S3 bucket. You become responsible for managing outdated artifacts, you are the one who has to account for multiple build plan branches, and much more. This is really a bit too much of overhead and it is extra annoying when you know this is a core Bamboo functionality and it’s supposed to work right out of the box.

Jenkins Remote Node

Read the Jenkins Remote Node installation post to get initial overview.

As you can see, there are common problems related to running SSH sessions as non-login user and others. I personally ended up running remote node as OS X Launch Daemon. This works, but is not ideal.

I have nothing yet to say about artifact copy speed from slave to master in regards to Jenkins setup, but stay tuned and/or post a comment if you have any knowledge on the matter.

Android

I wanted to mention Android in this last section, after all it’s mobile too.

I wrote one post about building Android - Build Android in The Cloud which already provides some details.

In regards to Bamboo vs Jenkins comparison, almost everything I mentioned for iOS applies to Android, except the very big and troublesome Mac OS X part. You can, and you should build Android on Linux systems, and that’s exactly what AWS instances are running on! It means that if you have 25 build agents/nodes in the cloud, all 25 of them can be used to build Android projects.

I had experience with building Android projects using Bamboo OnDemand setup hosted in AWS cloud. It worked (and what I was last told it still works) flawless and scalability of AWS really pays back. Despite the fact that building Android project was way more slower than building similar iOS project, that wasn’t really a concern given the number of resources available to do the job. I still had to run Android UI Automation tests (using Calabash) on Mac OS X box, because Linux agents in the cloud didn’t have enough horsepower to run Genymotion or Android Emulator reliably.

Summary

Despite my excitement with Jenkins Job DSL plugin there’s no clear winner for me.

Both solutions can be used for enterprise level mobile CI. Both can be used in pure UI-driven way while Jenkins also offers a code-driven approach with certain trade-offs.

I’d say, whichever server you use, whatever your CI requirements are, just make sure you take it seriously. In some way I’m calling out to all of you to help mobile CI to get out of its teen years.

Swift Dictionary Literal Convertible

2015-01-27T00:00:00+00:00

[OUTDATED]

A practical example of Swift’s DictionaryLiteralConvertible protocol application.

Note. This article content refers to Swift version 2.0 or earlier and is now outdated.

If “Literal Convertibles” sounds strange to you then a good place to start is an excellent NSHipster’s post.

OK, let’s say we have some struct in Swift.

struct Options {
  let timeout: Double = 1
  let message = "Timeout failed."
}

This is a structure holding timeout options for some abstract waiting function. If waiting takes longer that timeout then waiting is terminated and the message is displayed. For convenience default values are provided.

func wait(options: Options) {
  println("Waiting for \(options.timeout). Message: \(options.message)")
  // TODO: wait for some condition and fail after timeout
}

wait(Options(timeout: 0.1, message: "Custom timeout failed."))

Now, wouldn’t it be nice to add some syntactic sugar and pass a dictionary with key-value pairs for waiting options, instead of calling Options initializer?

wait(["timeout": 0.1])

Of course compiler doesn’t like this code, though it fails to tell us exactly why.

'Options' does not have a member named 'Value'

Let’s experiment a bit.

wait("SomeString")

Doesn’t make much sense but this time compiler complains with more sane error message.

Type 'Options' does not conform to protocol 'StringLiteralConvertible'

Basically it doesn’t know how to construct and instance of Options type from String.

So when we used dictionary before, compiler had no idea how to create an instance of Options from a dictionary, but sadly couldn’t communicate it back to us in a human-friendly form.

To be able to create Options objects from dictionaries Options has to conform to DictionaryLiteralConvertible protocol.

/// Conforming types can be initialized with dictionary literals
protocol DictionaryLiteralConvertible {
  typealias Key
  typealias Value

  /// Create an instance initialized with `elements`.
  init(dictionaryLiteral elements: (Key, Value)...)
}

To conform to this protocol we need to implement init and define type aliases for Key and Value. This was somewhat of a surprise for me, even when reading NSHipster’s post for the first time I didn’t realize that typealias are part of the protocol and had to be “implemented” (declared) too. I must have confused typealias with C’s typedef for a while, hence misunderstanding of typealias.

OK, first attempt in conforming to the protocol.

extension Options: DictionaryLiteralConvertible {
  typealias Key = String
  typealias Value = AnyObject

  init(dictionaryLiteral elements: (Key, Value)...) {
    for (key, value) in elements {
      switch key {
        case "timeout": self.timeout = value as Double
        case "message": self.message = value as String
        default:
        fatalError("Unknown key: \(key)")
      }
    }
  }
}

For dictionary keys the String type is used, and for values use AnyObject since we can pass either Double or String as a value.

I’m sure there’s a better more type-safe way to type alias Value to make it more “functional”. Probably using enums. I’m still learning though.

The init iterates over the contents of the dictionary and initializes struct’s members with values from the dictionary. The as is used to type cast AnyObject to Double or String.

I’m aware that use of self is not strictly required in this case and self.timeout = works same as just timeout =, but I tend to explicitly use self. syntax in initializers, kind of rule of thumb.

Let’s try it again.

wait(["timeout": 0.1])

It works. Also notice that we can now provide partial arguments, like only timeout but no message in this case, something we couldn’t with default struct initializer (would have to implement additional initializers).

Yet there are few things that smell in this code (apart from AnyObject I suppose).

Use of hard-coded string literals like “timeout” is not good, typos happen all the time and it’s quite painful to debug this type of errors.
Default case statement with fatalError plug.

Let’s now fix both these issues. First we will use a custom enum type for keys.

enum OptionKey {
  case Timeout, Message
}

Note that’s it not even backed up by a raw String type, there’s no need for that at the moment.

Now can rewrite the Options extension.

extension Options: DictionaryLiteralConvertible {
  typealias Key = OptionKey
  typealias Value = AnyObject

  init(dictionaryLiteral elements: (Key, Value)...) {
    for (key, value) in elements {
      switch key {
        case .Timeout: timeout = value as Double
        case .Message: message = value as String
      }
    }
  }
}

Main changes are

Use OptionKey for Key type alias.
Get rid of default: case, wrong key errors are now handled at compile time, not at run time.

Using the wait function is now as simple as

wait([.Timeout: 0.1])
wait([.Timeout: 0.1, .Message: "Custom timeout failed."])

You can grab the swift code and try it yourself in playground or just by running from command line with xcrun swift.

Swift and Objective-C Interoperability

2015-01-22T00:00:00+00:00

[OUTDATED]

I’ve stumbled on an issue while working on my hobby project. One of the few related to Swift and Objective-C interoperability.

Note. This article content refers to Swift version 2.0 or earlier and is now outdated.

Let’s setup a simple code base.

import AppKit
typealias BaseObjCClass = NSObject

enum PureSwiftEnum {
  case Value, AnotherValue
}

protocol PureSwiftProtocol {
  var value: PureSwiftEnum { get }
  var size: CGSize { get }
}

class ObjCSubclass: BaseObjCClass, PureSwiftProtocol {
  var value: PureSwiftEnum = .Value
  var size: CGSize = CGSizeZero
}

class GenericClass<T where T: PureSwiftProtocol> {
  var embeddedInstance: T
  init(embeddedInstance: T) {
    self.instance = instance
  }

  func accessValue() -> PureSwiftEnum {
    return instance.value
  }

  func accessSize() -> CGSize {
    return instance.size
  }
}

let objectGeneric = GenericClass(embeddedInstance: ObjCSubclass())

println(objectGeneric.accessValue())
println(objectGeneric.accessSize())

There’s pure Swift enum, protocol and a subclass of Objective-C class that adopts pure Swift protocol. By the way, have you ever tried declaring Swift enum with just one case?

Further on, I declare a generic class with type conforming to pure Swift protocol.

So what happens if I pass an Objective-C subclass to this generic class initializer?

To be honest, nothing happens most away. For reasons unknown to me my project compiled and ran. And it was running OK for a while until at some moment it started crashing consistently.

So the problem in this case is that I’m implicitly checking the conformance of Objective-C object ObjCSubclass to a non Objective-C (pure Swift) protocol PureSwiftProtocol. This check occurs when calling return embeddedInstance.value where embeddedInstance is an instance of ObjCSubclass but the access to it’s property happens by converting it to PureSwiftProtocol.

Apparently, this is a known issue. There’s a couple of discussions on StackOverflow (one, two).

Solution A: Back to Roots

The first solution is to turn PureSwiftProtocol into Objective-C protocol by using @objc notation.

@objc protocol PureSwiftProtocol {
  var value: PureSwiftEnum { get }
  var size: CGSize { get }
}

But that won’t make compiler happy because it has no idea how to map PureSwiftEnum into Objective-C. So you have to take it one step further. You have to declare PureSwiftEnum as Objective-C enum (with NS_ENUM), obviously you have to do it in Objective-C header file and properly setup bridging header in your project.

typedef NS_ENUM(NSInteger, PureSwiftEnum) {
  PureSwiftEnumValue,
  PureSwiftEnumAnotherValue,
};

Solution B: Wrap it Up

If you don’t want to revert back to adding Objective-C code with the hope that Apple eventually fixes this issue in the future, you can try another ugly trick - wrap your Objective-C class with pure Swift class that conforms to the same protocol.

class PureSwiftWrapper: PureSwiftProtocol {
  var objcInstance: ObjCSubclass

  init(objcInstance: ObjcSubclass) {
    self.objcInstance = objcInstance
  }

  var value: PureSwiftEnum {
    return objcInstance.value
  }

  var size: CGSize {
    return objcInstance.size
  }
}

Now you can pass an instance of PureSwiftWrapper to GenericClass

let wrapper = PureSwiftWrapper(objcInstance: ObjCSubclass())
let generic = GenericClass(embeddedInstance: wrapper)

println(generic.accessValue())

With this setup my crash went away and hadn’t reappeared since then.

Summary

Reference to objc-interop.swift to experiment and run it from command line or in Playground

Flappy Swift

2015-01-12T00:00:00+00:00

Yet another clone of Flappy Bird. Once again written in Swift. Accompanied by Flappy Boo written in Objective-C.

Shark 2 for iOS/OS X

2014-06-05T00:00:00+00:00

SHARK provides libraries for the design of adaptive systems, including methods for linear and nonlinear optimization (e.g., evolutionary and gradient-based algorithms), kernel-based algorithms and neural networks, and other machine learning techniques.

This script is created to build framework for Shark version 2.3.4.

To read the full article please follow the link.

Install Atlassian CLI with RPM

2014-05-30T00:00:00+00:00

Another follow up to introduction to Atlassian CLI.

This post describes how to create an RPM package for Atlassian CLI to install it on RMP-based Linux distributions.

Jump directly to Summary if just want to grab the end result.

Why Bother?

Same question as one would as for using Homebrew on OS X. And same answer again - Automation.

Let’s assume you already use Atlassian CLI Client for number of build tasks, like automatically updating JIRA tickets, Confluence pages, Stash pull requests, etc.

Let’s further assume that your company has a proper CI environment, for example RPM-based Linux instances running in AWS cloud. Each instance runs one build agent (slave, node or whatever you call it).

One of the reasons going into the cloud is scalability, you can go from one to few dozens of identically cloned agents with literally one click of a mouse.

Normal practice is to refresh those instances repeatedly, e.g. on a 2 weeks schedule, and refresh in this context means recreating them from scratch. This is when an updated OS image is used for new instances, you can put new updates and packages into this new image to make it available to all build tasks by default without additional installation. For example, you could not just have RVM installed, but also install few most used rubies for 2.0 and 2.1.

Preparing new image has a funny name, that is Baking (remember Brewing?). What’s amusing, the whole baking process is done by the very same CI setup it will be applied to. Keeping an eye on the whole infrastructure is actually quite a challenge and you would most likely have DevOps or DevSupport team to look after it.

In any case, to conclude this long passage, one of the reasons to have an RPM package for Atlassian CLI is to be able to bake it into your build agent.

Create RPM Spec

Creating and RPM package is very similar to creating Homebrew tap, only in this case instead of Formula you need to create RPM Spec.

Let’s put first lines into the spec

First three lines are a result of googling as well ans trial and error process. I’m not an experienced RPM package builder, so there are things that I will have to leave unexplained.

Next lines are self descriptive. The %{varname} syntax is the way you can pass variables into RPM spec when using it with rpmbuild, for example

Some of the header attributes worth mentioning separately. For starters the Source: attribute should point to your source tarball. But it doesn’t have to be a local file, it can be a URL just like Homebrew’s url attribute. You can then download and unzip it with one call to %setup macro.

I myself just found out about it recently and didn’t try it yet. So in this article I’ll do the job of setup macro with shell commands, but this is definitely where improvements should be made.

Then there’s _tmppath variable. You will notice later that it’s not passed to RPM script directly anywhere, instead, it’s picked up from a special .rpmmacros file. You’ll see how it’s used later on.

Install

Now it’s time to define install command. It’s again very similar to Homebrew’s install. So I will give here brief description of the steps with the code and for more details you can always get back to Homebrew post.

Prepare and Setup

Since we don’t use %setup macro, we have to some of the work here. Remove old build directory, unzip the source tarball and cleanup Windows stuff right away.

Patch and Move

Next is patching time and once patching is over move the files to a proper location (bin folder). I’ve explained why this is required in the post related to Homebrew formula, so have a peek there.

When patching we’ll update all the .sh scripts and put new relative path to JAR files, we’ll also insert proper JAVA_HOME export in each.

Then it’s time to customize all the Atlassian product URLs as well as put a proper service account username and password if you plan to save keystrokes in the future.

Finally rename ambiguous all.sh to atlassian-all.sh and move all .sh files to bin folder. I personally prefer to drop .sh part from the filename in the process.

Files and Symlinks

Now it’s time to tell RPM spec which files are part of final installation.

In post-install stage we want to symlink all the shell scripts and JAR files to a corresponding directory in /usr/local. The reason behind that is because /usr/local/bin is already on our PATH (if not, then add it as described in Homebrew post).

We also add info for post-uninstall process so it can remove all these symlinks for us.

Clean

No comments on this one.

Changelog

Add some change log and you are done with the spec.

Build RPM

It’s time to build an RPM based on the spec we have just came up with. In this example I’ll use Makefile which helps to present material in more simple and organized way than just shell scripts.

Define all the basic configuration, like version, release, Java version, etc. Note the use of noarch here for architecture. We are working with collection of JARs and shell scripts, there are no sources files that need any compilation at all, so we specify that we are not building for any particular architecture.

Now define step by step what needs to be done using Makefile targets (aka recepies).

Download the package if it doesn’t exist yet.

Then unzip. This place partially explains why I avoided %setup macro. Setup macro assumes source is a tarball and uses tar utility to unpack it. But with Atlassian CLI the source is just a .zip file, so setup macro generates commands that don’t work.

Once unzipped, we also link symbolically versioned folder to ${PACKAGE} file. This makes it easier to write next steps.

Now pack unzipped file into a tarball. This is all due to specifics of rpmbuild, it want’s tarball and we have to comply. Put the tarball into distributive directory.

Finally we can build RPM.

Still we end up doing some additional work here. rpmbuild expects a certain directory structure in it’s root build folder. Once again, we’re pretty much doing the %setup job here.

We will build an RPM in ~/rpmbuild directory, so we create one with number of required subdirectories (all those names in caps and tmp).
Then move tarball to SOURCES directory
Copy our RPM spec to SPECS
Then add %_topdir and %_tmppath to a special ~/.rpmmacros file. rpmbuild will scan ~/.rpmmacros and pass all picked up values to RPM spec when building it.
Finally call rpmbuild passing version, release and all the other vars.
- The -bb option tells rpmbuild which steps to execute.
Once rpmbuild is done, copy RPM package to distributive directory.

So, are you ready to try it?

I ran it on Fedora 20 as well on a custom Linux distribution running in AWS cloud. Since the package does not depend on architecture, you could in theory build it on OS X machine, but it’s not what I would recommend, getting proper rpmbuild port configured is not something you enjoy very much.

Install RPM

To test your installation, use these commands

Now you can hand RPM package over to your Dev Support guys, they’ll put it into local repo and make RPM install a part of bake or post-bake process.

Yet nothing stops you from creating a CI plan (job) for the Atlassian CLI RPM package itself. You can run make rpm and test rpm install on the very same build agent this package is targeted for, thus creating yet another “CI Loop”, which is good.

Summary

Create RPM spec
Run make rpm -f RPMMakefile using this RPMMakefile

Install Atlassian CLI with Homebrew

2014-05-30T00:00:00+00:00

This is a follow up to introduction to Atlassian CLI.

In this post you will learn how to create custom Homebrew formula, install Atlassian CLI via Homebrew tap and customize tools for you company environment.

If you don’t care about all the details and steps involved, you can jump right to Summary section.

Brew

Homebrew (aka “brew”) is a missing package manager for OS X. After you install it, you need to update your PATH. By default /usr/local/bin is not in the system path, so modify your ~/.bash_profile before you continue.

Why Brew?

A perfectly valid question. Why would you go into all the trouble of creating formula, when you could just unzip, copy and update PATH, or even write a simple shell script to do that.

Well, Homebrew does all that and then much more. After initial installation upgrading the tools is as easy as brew update && brew upgrade. It also makes installation and upgrade process easier to other people in your organization. And finally, it’s just cool.

OK, Let’s Brew!

Create Formula

You could use brew create <link> command, but that will create formula in /usr/local/Library/Formula. Instead let’s create it manually.

Let’s say our company is called i4nApps (I know, it’s a weird name…), so create new Ruby file

With contents like this

These are the basics of brewing.

Your custom formula class needs to subclass Formula class.
version in our case is “3.9.0”, that’s the latest Atlassian CLI Client version at the moment of writing this post.
homepage points to Atlassian Marketplace page.
url is used to download source code. Note a bit of tweaking at the end of the link #{version.to_s.delete('.')}, that’s to remove dots.
sha1 obviously is SHA-1 calculated for file downloaded from url.

install method is where you do all the installation magic once the source is downloaded and unzipped.
test is used to test formula after installation. Normally you just execute main program installed by the formula, in our case it can be jira.

Now let’s take advantage of the fact that Homebrew formula is just a Ruby class and add few more custom lines to use later.

release will be used for managing multiple releases for same version
java_version will come handy for setting JAVA_HOME environment variable. Default is “1.6” but you can user JAVA_VERSION environment variable to override default settings.

Atlassian CLI Client Compatibility page claims that “Client requires Java 1.6 (recommended) or above.” I have had problems using it with Java 1.7, so in this guide we’ll stick with 1.6.

`install`

Once execution enters install method, the source is already downloaded and unzipped. Things that you do in install method occur in temporary directory created by Homebrew.

This is the place to run things like ./configure and make. However in our case we have no source code to compile. Atlassian CLI is basically a collection of JAR files with shell wrappers for them.

Our job is to cleanup first, then patch and rename some scripts and finally move the whole bunch to /usr/local. /usr/local is also called prefix, this is the location where Homebrew installs all packages, additionally there’s a special prefix variable available in the formula.

Cleanup

So let’s cleanup. Since we are installing on OS X, we don’t need all the Windows stuff.

As an improvement, this could be a good place to remove examples.

Patch and Move

Now it’s the time to mention one of the things done wrong in Atlassian CLI tools package. There’s a reason for having all those folders like lib, libexec, etc and so on. In Homebrew world each of those folders serves a special purpose. Yet the most important folder of all is missing, that is bin!

Remember that part where you added /usr/local/bin to your PATH? When Homebrew installs a package it copies things from temp folder to prefix (/usr/local), bin is copied too. Every executable in the bin becomes available system-wide because of the way you updated PATH.

In case of Atlassian CLI, all the shell scripts are sitting in the root folder, bin is not there at all. This needs to be fixed. We will do it in 2 steps

Create bin folder and move scripts to it
Patch the scripts

The order doesn’t really matter. Patching is required because each shell script is a wrapper around JAR file and contains relative path to that JAR. So if you move the shell script, you have to update the relative path as well.

Here we replace relative path to lib with ../lib. We also set JAVA_HOME here using java_home OS X utility and java_version method. This is to be sure Java version is as we expect it.

We just moved all the .sh scripts to bin folder. I also prefer to drop the .sh part. Finally all feels too ambiguous, so rename it to atlassian-all.

Install to Prefix

We can finally install everything to prefix (/usr/local), that’s where prefix variable is used

Customize

This part is optional. Unless you customize the scripts you will have to use --server, --user and --password switches each time you call commands, this is to provide server, username and password.

Of course this can be solved with aliases, but then you’d have to configure aliases on each build box. Anyway, the developers of the CLI package offer you another solution. As I said, this part is optional, if you don’t need customization you can skip to next section.

The atlassian.sh (which we renamed to atlassian and moved to bin) is there for customization. Have a look inside that file

This is where you can customize your Atlassian products username and password, as well as additional JVM settings. That’s usual practice for organizations, you have a special user account (service account) than can access the whole range of products with single username and password.

And there’s another block of code like this, which is used to customize Atlassian products server urls.

You will need to replace all the https://***.example.com with your company urls. If you have multiple instances of same product in your organization, you can add another elif block for that. For example you are in the middle of migration from Confluence server https://confluence.example.com to a new instance https://confluence.ni.example.com, for a while you want to be able to use both, so add another block like this

In this example there won’t be multiple instances for same product, but it would be possible to customize scripts to handle that case as well.

So let’s write some Ruby again. For the company called i4nApps we will create a nested class I4nAppsEnv that will contain all the company specific environment settings.

username and password methods pick up values from ATLAS_USERNAME and ATLAS_PASSWORD environment variables. Set those when running brew install or brew upgrade.

Next the server method returns server url for each product type.

Finally patch method patches the shell script the way we want it.

Replace default username and password with values provided via environment variables
Replace all the server urls with your company urls

You have to add one more line to install method in the formula

Make sure to put this line before atlassian.sh is moved and renamed.

The nested class is used because Homebrew only expects one formula file. If you have other files used as an external dependencies, Homebrew will not fetch them from repository when running tap command.

Push to SCM

We are done with the formula. It’s time now to push it to a repository. Whatever is you favorite SCM - use it. In this example we will use Git repository hosted with Stash. For example

ssh://[email protected]/mobile/i4napps-atlassian-cli.git

Tap, Install and Upgrade

Tap

One of the ways to install Homebrew packages from custom repositories is to use Taps.

There’s a number of rules to follow when creating your taps. That mostly matters if you plan to share this formula with the rest of the world. For internal use in your organization you can neglect some of these rules.

But there’s a trade off as well. Since we didn’t name the tap repository properly, we won’t be able to use tap command like brew tap username/repository. Instead we will do same actions as tap does only with few lines of shell script.

brew tap clones the repository from GitHub and puts it into the taps directory. This is how you can do it directly

Install

Now you can install, this part is simple

Upgrade

To upgrade you need to update brew repositories, including the taps, then upgrade the specific package.

Summary

Create formula Ruby file and put it into a repository

i4napps-atlassian-cli.rb

Create a Homebrew tap, install and upgrade

Share Xcode Schemes

2014-05-29T00:00:00+00:00

So you are facing one of these Xcode errors where it tells you that there’s no such scheme? This post could help to explain why Xcode does it and how to solve this problem.

Shared vs User Schemes

While setting up build server for iOS app I have faced this issue multitude of times. Xcode schemes can be either shared or not. By default schemes are not shared and are owned by a user that creates them.

Here’s how it looks in Xcode, note the last column with checkboxes.

If you look inside kartoteka-reloaded.xcodeproj folder you will see how Xcode stores the schemes. Here’s how it looks when kartoteka-reloaded.xcscheme is not shared

kartoteka-reloaded.xcodeproj/
├── project.pbxproj
├── project.xcworkspace
│   └── contents.xcworkspacedata
├── xcshareddata
│   └── xcschemes
└── xcuserdata
    └── grebenetsm.xcuserdatad
        └── xcschemes
            ├── kartoteka-reloaded.xcscheme
            └── xcschememanagement.plist

Now let’s tick the “Shared” checkbox then list directory contents again

kartoteka-reloaded.xcodeproj/
├── project.pbxproj
├── project.xcworkspace
│   └── contents.xcworkspacedata
├── xcshareddata
│   └── xcschemes
│       └── kartoteka-reloaded.xcscheme
└── xcuserdata
    └── grebenetsm.xcuserdatad
        └── xcschemes
            └── xcschememanagement.plist

Notice how the kartoteka-reloaded.xcscheme moved from user data folder to shared data folder. This is basically what makes scheme a shared one.

The general practice for Xcode projects .gitignore file is to ignore user data

# .gitignore
xcuserdata/
*.xcuserdatad

So when you check out source code on a build box, there won’t be any user schemes inside .xcodeproj folder and xcodebuild won’t be able to see the schemes and will fail to build them.

Solution

You can solve this problem either manually or automatically.

Of course you can just talk to devs and ask them to share the scheme. Done! You can even do this change yourself and create pull request with changes.

But this approach will not work in some cases. For example, if you want to run UI automation tests with Calabash, the steps are

Duplicate existing Xcode target and name new test target with -cal suffix
Add Calabash framework to test target
Build -cal test target and run tests

The first step is done with calabash-ios setup command. When new target is created a scheme is created for it as well. This is the default setting for all Xcode projects and in this post we’ll assume that’s the way you have it configured as well.

Now the tricky part, it doesn’t matter if original scheme was shared, the new -cal scheme will not be shared. That means you won’t be able to build it from command line.

Since it all happens on a build box as part of a build plan, you can’t push anything back to the repository, you have to find a way to make this new scheme shared right now.

The answer to your problems comes from Ruby world. In particular the xcodeproj Ruby gem. This is an incredibly handy library to work with Xcode projects and workspaces. You can do pretty much anything you need, create and modify targets and schemes, add new files to targets, modify build settings and other properties, and, of course, share schemes. By the way, xcodeproj is used by CocoaPods and that says a lot.

Go ahead and install the gem

[sudo] gem install xcodeproj

Now create a simple Ruby file, name it whatever you want

#!/usr/bin/env ruby
# share_schemes.rb

require 'xcodeproj'
xcproj = Xcodeproj::Project.open("MyProject.xcodeproj")
xcproj.recreate_user_schemes
xcproj.save

This is it! Put your Xcode project name in there, then run and the scheme will be shared.

chmod +x share_schemes.rb
./share_schemes.rb

Caveats

It sounds to good to be true, right? There’s a number of situations where sharing a scheme via Ruby script will not work as expected.

If your Xcode project already has a shared scheme, then you will end up having one scheme from user’s data directory and another one form xcshareddata. Xcode IDE will pick up both and that’s the reason why you see same scheme twice with project name in parentheses.

That’s not very bad and doesn’t normally cause any problems. Until that moment of time when you modify one scheme and forget about another. The best way to avoid this problem is to share schemes from day 1, Xcode will not create user schemes then.

The real trouble begins if you didn’t have any shared schemes and the scheme that you want to recreate and share is linked to a test target. That’s the default configuration for unit tests. So the problem is that xcodeproj doesn’t recreate dependencies to test target. If you run a xcodebuild test action you’ll be surprised to see it failing. Unfortunately there isn’t an easy workaround for this problem, so you’d better share those schemes manually and commit changes to source control system.

Summary

Surely the Ruby script can be improved, you’d want to pass Xcode project name as an argument or even look it up automatically.

As usual, in Summary I just provide file listing with a solution ready to copy-paste. Here’s more advanced Ruby script for your use. You can just get the file directly if you’d like, share-schemes.rb

#!/usr/bin/env ruby
# share_schemes.rb

require 'optparse'
require 'ostruct'
require 'rubygems'
require 'xcodeproj'
require 'colorize'
require 'fileutils'

# Option parser
class OptionParser

    # Parse options
    # @param [Array<String>] args command line args
    # @return [OpenStruct] parsed options
    def self.parse(args)
        options = OpenStruct.new
        options.project = nil

        opt_parser = OptionParser.new do |opts|

            opts.banner = "Usage: #{File.basename($0)} [options]"

            opts.separator("")
            opts.on('-p [PROJECT]', '--project [PROJECT]', "Xcode project path. Automatically look up if not provided.") do |project|
                options.project = project
            end

            opts.separator("")
            opts.separator("Help:")
            opts.on_tail('-h', '--help', 'Display this help') do
                puts opts
                exit
            end

        end

        opt_parser.parse!(args)
        options
    end # parse()
end

options = OptionParser.parse(ARGV)

# Lookup for Xcode project other than Pods
# @return [String] name of Xcode project or nil if not found
def lookup_project
    puts "Looking for Xcode project..."
    # list all .xcodeproj files except Pods
    projects_list = Dir.entries(".").select { |f| (f.end_with? ".xcodeproj") && (f != "Pods.xcodeproj") }
    projects_list.empty? ? nil : projects_list.first
end

# lookup if not specificed
options.project = lookup_project if !options.project
if !options.project then
    puts "Error".red.underline + ": No Xcode projects found in the working folder"
    exit 1
end

puts "Using project path: " + "#{options.project}".green

xcproj = Xcodeproj::Project.open(options.project)
xcproj.recreate_user_schemes
xcproj.save

Right, this is one of those cases where actual meaningful code is very small (just 4 lines), the rest is options parsing and error checking, but then it’s worth it in the end.

Finally run it

chmod +x share_schemes.rb
./share_schemes.rb -p "MyProject.xcodeproj"

P.S.

Related thread on StackOverflow.

Build Android in the Cloud

2014-05-29T00:00:00+00:00

In this example, you are given a task to build Android app on a 64-bit AWS Linux build agent running in Amazon Cloud (AWS).

Install Android SDK

You have to start with installing Android SDK first.

Download

Start with downloading latest Linux version, found on this page. You can find direct link to tarball in “DOWNLOAD FOR OTHER PLATFORMS” section.

wget http://dl.google.com/android/android-sdk_r22.6.2-linux.tgz

Extract and Move

Extract it and put in preferred location, in this example it’s /usr/local/opt/android-sdk

# Extract.
tar xzf android-sdk_r22.6.2-linux.tgz

# Move.
mv android-sdk-linux /usr/local/opt/android-sdk

Configure `PATH`

Now configure ANDROID_HOME and update the path. That is done by modifying ~/.bash_profile, create the file if you don’t have it yet.

export ANDROID_HOME=/usr/local/opt/android-sdk
export PATH=$PATH:$ANDROID_HOME/tools
export PATH=$PATH:$ANDROID_HOME/platform-tools
if [[ -d $ANDROID_HOME/build-tools ]]; then
    ANDROID_BUILD_TOOLS_HOME=$(dirname $(find $ANDROID_HOME/build-tools -name aapt | tail -1))
    echo "Android Built Tools located: $ANDROID_BUILD_TOOLS_HOME"
    export PATH=$ANDROID_BUILD_TOOLS_HOME:$PATH
fi

You noticed that adding Android Build Tools to the path is optional. That’s because we didn’t install Build Tools yet.

Update Android SDK

This is where you update Android SDK, install all the tools and APIs, including Build Tools and Android Support Repository and Library. Remember, this Linux instance is running in the AWS Cloud meaning it’s headless (no GUI) and you only can work with the shell.

If you happened to read older version of this article, you’d remember seeing a lot of shell scripts using grep.

This is a revised version, with much simpler and cleaner code.

The command that does the job is android update sdk. You need to use --filter option to specify list of packages you need to update or install. To get readable identifiers of all available packages, run the following command

android list sdk --all --extended

--extended flag is used to display extended information about each package, including a human readable identifier. --all is needed to include extra packages, like build tools, by default they won’t be listed. Here’s an example output.

----------
id: 3 or "build-tools-20.0.0"
     Type: BuildTool
     Desc: Android SDK Build-tools, revision 20
----------
id: 4 or "build-tools-19.1.0"
     Type: BuildTool
     Desc: Android SDK Build-tools, revision 19.1
----------

You can now compose filter as a comma-separated list of all package identifiers you want to install. Since it’s a headless build box, you will have to use --no-ui option, and --all is needed to include extra packages.

FILTER=tool,platform,android-20,build-tools-20.0.0,android-19,android-19.0.1
android update sdk --no-ui --all --filter $FILTER

This example installs Android SDK Tools, Platform tools, Build tools versions 20.0.0 and 19.0.1, as well as SDK Platform 19 and 20. Customize the list to your needs using proper identifiers.

Answering the Prompts

But it’s not done yet. When installing or updating, you will have to accept license prompts. Each package can have it’s own license requiring you to answer a prompt with “y”.

You’d probably think of yes command line utility designed for this particular task. However it will not work. yes outputs “y” to stdout too often with no options to put delays in between. Android SDK update tool expects not just “y”, but a “y” followed by return key, in other words, it expects “y\n” string as a whole. I don’t know the exact mechanics of yes command, but if you try something like this

FILTER=tool,platform,android-20,build-tools-20.0.0,android-19,android-19.0.1
yes | android update sdk --no-ui --all --filter $FILTER

you will see that the license prompt will complain about incorrect input and fail after a number of attempts.

The solution is to put certain delay between “y” outputs to stdout. This code I found on the web, it’s reliable and does the job.

FILTER=tool,platform,android-20,build-tools-20.0.0,android-19,android-19.0.1
( sleep 5 && while [ 1 ]; do sleep 1; echo y; done ) \
    | android update sdk --no-ui --all \
    --filter ${FILTER}

Project Specific SDK Update

If you have various Android projects, each with it’s own requirements for Android packages, it would be reasonable to add Android SDK update task to the project’s build configuration. In this example I will use Gradle, so here’s an example of Gradle task

task updateSDK(type: Exec) {
    ext.filter = "tool,platform-tool,android-20,build-tools-20.0.0,android-19,build-tools-19.1.0,build-tools-19.0.1,extra-android-support,extra-android-m2repository,extra-google-m2repository,extra-google-google_play_services,extra-google-google_play_services_froyo"
    commandLine "sh", "-c", "( sleep 5 && while [ 1 ]; do sleep 1; echo y; done ) \
    | android update sdk --no-ui --all \
    --filter ${filter}"
}

Caveats

The last and very annoying bit, is that this script seems to update packages even if they are already installed. At the moment I have no clue what’s causing this behavior and what’s the best workaround. When it comes to running this script on one of the Bamboo agents in the cloud it doesn’t really matter, but when the script runs on one and only Mac build agent you have - this really slows down each build.

Install 32-bit Libraries

Before you try to build anything, you have to do one more thing.

Remember, the host OS is 64-bit Linux system. Android requires a bunch of 32-bit libraries and you have to install them. The Linux system in question is RPM-based so this example uses yum command, change it to apt-get or another package manager specific for your OS.

# Install 32-bit libraries.
sudo yum install glibc.i686, zlib.i686, libstdc++.so.6 libz.so.1

Now you’re good to go!

Summary

This is a TLDR or a summary section, that simply lists the solution “as is”.

# Install 32-bit libraries.
sudo yum install glibc.i686, zlib.i686, libstdc++.so.6 libz.so.1

# Update Android SDK on headless server.
FILTER=tool,platform,android-20,build-tools-20.0.0,android-19,android-19.0.1
( sleep 5 && while [ 1 ]; do sleep 1; echo y; done ) \
    | android update sdk --no-ui --all \
    --filter ${FILTER}

Atlassian CLI Client

2014-05-29T00:00:00+00:00

If you happen to use Atlassian products like JIRA, Bamboo, Stash, Confluence, etc., you may be surprised to know that all these products are backed by a handy command line interface, namely Atlassian Command Line Interface.

Imagine opening, modifying and closing JIRA issues from command line. What about starting Bamboo build plans, opening and closing Stash pull requests, updating Wiki pages?

“Why would I do that?” you would ask. Indeed, there’s not much benefit to automate these things for you as a user, unless you are some bash addict. However, stop for a moment and think “Automation” and you’ll see how many things a basic shell script can do. All you need is Java and shell interpreter, which are present on any decent build box.

Atlassian CLI Client is available on Atlassian Marketplace. It is not developed by Atlassian though, which is very important to know if you plan to use it.

This is where the confusion between Atlassian CLI Plugin and Atlassian CLI Client needs to be cleared up.

Plugin is not free and should be purchased for Atlassian products, e.g. there’s CLI plugin for JIRA, Bamboo, Stash. This basically a UI for command line tools.

Client (command line tools) can be freely downloaded and installed. You can find a few outdated Homebrew formulas on GitHub that automate client installation. So you might easily assume that it’s free, but it’s not!

You must purchase a license at least for one of the Atlassian CLI Plugins in order to use the Client.

The purpose of this post is to introduce you to Atlassian CLI Plugin if you don’t know about it yet. The plugin is very well documented and comes with tons of examples.

The thing that I didn’t like about it is installation process. It’s a straightforward “download, move, update PATH” process. There’s nothing bad in the process itself, but when you need to roll out CLI tools on a dozen of build agents, you start looking for a better way.

So I follow up this post with 2 more

Install Atlassian CLI with Homebrew (OS X)
Install Atlassian CLI from RPM (Linux)

NSBogan

TeamCity Kotlin REST Client

⚙️ Setup

🔨 Implement

👷‍♂️ Build

🏃‍♀️ Run

Xcode Build Settings in Depth

🗄 Background Check

⬇️ Level Down

⚙️ Xcode Specs

🔘 Options

Types

ℹ️ String and Path

ℹ️ StringList and PathList

ℹ️ Enumeration

ℹ️ Boolean

🗺 Command Line Flags Mapping

ℹ️ Command Line Arguments

Scalar Types

List Types

Enumeration Type

Boolean Type

ℹ️ Command Line Flag

ℹ️ Command Line Prefix Flag

ℹ️ Additional Linker Flags

References

Conditions

Other Properties

👩‍🏫 Example

👷‍ Application

Fastlane for Enterprise

Enterprise Environment

Whitelist

🤖 Service Account

Using Fastlane behind the Proxy

Curl Test

🔏 Root CA and SSL Certificates

💲 Environment Variables

🚎 Transporter

Installing Transporter

Patching Transporter

Transporter and Proxy

Another Proxy

Upload ipa

Fastlane Session

Summary

Xcode Build Phases and Environment

The Problem

The Example

Login Shell

Build Phase Scripts

Tips

Script Errors

Managing Build Phase Scripts

Swift Refactoring - Summary

Swift Refactoring - Debugging

Ninja

Xcode

Swift Refactoring - Implementation - Part 2

Swift Refactoring - Implementation - Part 1

Add Refactoring Kind

Is Applicable

Find Nested If Statements

Walk the Tree

Test

Swift Refactoring - Tests

Test Refactoring Kind

Cursor Position Reference

Test Refactoring Transformation

Running Tests

Swift Refactoring - Setup

Latest Xcode

Stable Source

Building Xcode Toolchain

Swift Refactoring - Intro

P.S.

Async Swift Scripting

Build Alamofire

CocoaPods

Carthage

Upload `ipa`