Working with external teams to ensure that they fit your requirements into their schedule

I work in an organization that is fairly large, being a $500 million organization, having a number of products being worked upon by different product teams. As a part of this process, there are also many common components that are made by a few central teams, which provide functionality common to most of the products. For example, when you consider that functionality such as reading and writing to optical media, being able to open and write to video files, etc are common functionality, it makes a lot of logical sense to ensure that these functionality are done by a common team and then other teams pick it up. It is not such a open and dried system though, there are elements of complexity in these systems though:
- The various product teams that pick up these common components or code have different schedules, while the common component teams follow a set schedule (in most cases, this schedule is set primarily by the most important product team - for example, if you take the case of Microsoft, the MS Office or the Windows team will have a much higher priority than somebody working on Skype, and the schedule for a common component will be more geared towards the schedule of Office rather than that of Skype)
- The features needed for each product could be different, even if these are tweaks. The common components may provide a core technology, but even in these, there could be significant differences. A product that is geared towards an expert in the field of video would demand support for a more comprehensive set of video formats, while a more entry level product would just want support for a few widely used video formats. The makers of the common component have to work towards balancing the needs for such different requests for requirements, using the priority of the products that are requesting such features as a variable in making decisions.

As a result, if you are not part of the most important team, you have to keep in mind that your requests, even though important to your team, and critical, may not make it to the final list of items that the component team is working on. We have been in that position, and it can be pretty painful, but you are reminded that it is a decision meant for the good of the organization. What can you do about such a situation ? Well, there are a number of items that you can do:
- When you know that such a prioritization problem can happen, you need to ensure that for requirements which require a modification in the common component, you start working a bit early on those requirements and have the relevant discussions with the component team early. The later you present your requirements to the external team, the more problematic will be the conflicts that the external team will have to factor in, and the higher the probability that your requirements will not make it.
- Strike up a discussion with the product team that drives the requirements of the component team and learn about what are some of the new features that they are looking for. Evaluate those to see which of those fit into your requirements, which will give you a higher chance of ensuring that your features do make it.
- Even if the requirements differ like the point made earlier about different products requiring different levels of video format support, getting into discussions early and having more discussions geared towards such kind of support can ensure that the component team is able to provide alternative functionality, such as the same component providing different levels of functionality depending on variables that are passed to the component.

Synchronizing multiple activities together in a complex environment

Typically, during the software development cycle, most of the activities are straight-forward. For most features, the way that the cycle flows is that the feature request flows into the requirement phase where the requirements are defined (if you are doing Scrum, then there are User Stories defined for the feature that make up the feature request), these are then converted into design, and then into the coding phase followed by testing. The sequence of activities is typically linear, and not complicated, the only variables being the amount of resource effort needed for the work, and the schedule of the feature.
However, sometimes you come across a feature that is more complicated than the normal case. There can be complex features which have dependency on multiple other features, and it is takes effort on the part of the team, and of the project manager to work on these dependencies, figure out their schedules and ensures that everything in in place and at the same time, things are going on track.
For example, consider the case of a feature which depends on a change in the database structure (addition of one more table to the database structure as well as modification of an existing table) as well as hook-ups with other features (to send and take data from this new feature). In addition, since this is an existing product and a new version is being developed, users who more from the previous version of the product to this new version will need to ensure that their existing data is migrated from the previous database structure to the new structure. For this purpose, there will need to be data migration scripts that are ready, which will need to be run before testing of the new feature can happen.
Typically in most teams, this work is done by separate feature teams. All changes to the database structure (there may be changes requested to the database from multiple feature teams and it is more efficient to ensure that only one team is doing the required database changes), and similarly, other feature teams also need to make the changes on their end.
How do you ensure that all this is happening properly ? For the feature to work properly, it is necessary that all these changes from the different feature teams be synchronized to happen so that the feature is done. This requires intensive work from the project manager to work with the schedules of these different teams, shuffling them around until these different tasks all are planned to happen before the main feature. This can be captured in a tool, where the required tasks are all captured as dependencies of the main feature, thus ensuring that the team working on the main feature is not expected to start work before the dependencies are all done (or atleast, are done to the required degree as per the dependency logic). This also ensures that any modification to the schedule of these tasks are captured in the tool, and in turns affects the schedule of the main feature.

Working with external teams for getting them to resolve their defects

When we do our planning for the next release of the software, we normally consider a number of risks, seeking to define a risk database that tabulates various known issues that could cause problems to the schedule and the smooth progress of the project. One lesser known risk that sometimes does not make it to the list of known risks relates to defects with people who are outside the core development team. For example, you may have a component that is being used in the product, and there is a defect against that component. Such defects have a higher risk than defects that are with the core team, even if the defect is of the same severity. There are many reasons for these defects having a higher risk. Some of these are:
- The outside team may not follow the same schedule or the priority as the product team.
- In fact, supporting the product team may be a lower priority for the component team, since they may have been given a task of providing support to other teams. This can happen a lot in organizations where there are a number of teams, and a strategic product will get much more support. For example, if the MS Access team and the MS Office team report defects in a component used by both teams, who do you think will get the priority support. In such cases, there will need to be more discussions, escalations, and other such measures before there is support granted.
- There is a much higher degree of coordination and communication needed when interaction with a developer who is outside the core development team
- The same defect may be treated as expected behavior by another team that is using the same component. You may be having a doubt about why a defect can be treated as a feature, but in some cases, a particular workflow behavior may be seen as a defect by one team, and desired behavior by another team. In such cases, the situation may be such that the defect may never be fixed.
- When the defect is with a component that is an open source component similar, there is no guarantee that such a defect may be fixed.

Given some of these reasons above, defects with developers who are outside the core development team are a much higher risk than most teams tend to visualize. We once had a hair raising case where the component we were integrating had a defect, but somehow the developer working on it was not able to replicate the defect. This was close to an important pre-release milestone, and we needed to get some fixing done on this defect, but this back and forth discussion and coordination at a remote location meant that it took around 2 weeks to get the defect fixed and a new component integrated, and this was a big problem for us.
What do you do when you have such dependencies on extended teams ? Well, you can never really get away from the defects, but atleast have processes setup to reduce the time period and coordination cost for these defects. What should you do ?
- Have an understanding with the remote team about the level of testing at their end. If necessary, send them test cases and ask them to provide completed test cases before they provide the component. This helps in reducing the chances of defects passing through their end.
- Setup a process where they can received updated copies of your software so that they can test with such latest software
- Ensure that the external team totally understands your defect management system and processes and they have been provided permissions for the software.
- Have a protocol with them about exchanging information with them about defects and if there are queries on these defects
- Have a regular meeting with them, this help in ensuring that any issues from either end are resolved quickly
- Ensure that any serious defects are communicated to them at a higher priority. This also includes defects that need to be fixed on priority.
- Define an escalation matrix with them, so that if there are cases where you are dis-satisfied or need quick action, what is the process to follow.

All of these will not mean that you will not have external dependencies on defects with external team members, but the problems associated with these defects will be reduced.

Working with people who are not so responsive as developers or testers ..

It can really test the patience of a team when they work with people who do not seem to follow the same guidelines, processes and schedules as the rest of the team does. It seems a bit odd though. The schedule is the most important item of a software development project, with a lot of effort going towards determining whether the team is following the schedule or not, and if there are any slippages from the schedule, it can imperil the success of a project. If there are slippages of the schedule, then it would take a lot of effort from the project manager and other managers and senior members of the team to get the project back on track. In such a case, it seems strange that there can be members of the extended team who are not so committed to the schedule (actually, it is not right to say that they are not committed to the project, it is just that it can be a challenge to get them to follow the details of the schedule).
Who are these members of the extended team ? Well, let me lay out a few candidates (and this does not mean that every one of these roles will not follow the details of the schedule, but I do know many people who were in such roles with whom it was a challenge to get them to follow the schedule) - these are typically the more creative people of the team - in some cases, the Product Manager; in more cases, the UI Designer or the Experience Designer; and even in other cases, some of the more senior members of the development team.
Does this really happen ? Well, yes, it happens like this all the time. Let me layout some examples - during the course of a project schedule, there is an initial time period when the requirements need to be defined by the Product Manager (with a certain detail inherent in these requirements, a level of requirements that is enough for the workflow designer and the development team to do a certain level of estimation), and more often than not, unless I, as the Program Manager would do a lot of reminder to the Product Manager, there would always be some amount of delay, or the requirements that were available were not of enough detail. As a result, by a couple of cycles, we had actually started giving a buffer so that after all the urging, there would be time to do a couple of cycle of the requirements. Given that the Product Manager is also typically a senior person, we did not try any other method of ensuring that they finished their work by the scheduled time; instead we added a buffer of around a week in the overall schedule.
The bigger problem was when we were dealing with the experience designer / UI designer. The interaction with this person was on a regular basis, for more than 70% of features (given that most of the features needed some kind of UI work, or needing some kind of workflow optimization). Hence, it was not only the overall schedule, but also the schedule for each feature that needed dates and details for work supposed to be done by the UI designer. The work done by the designer followed in a logical order to the requirements and was needed for the development team to do their work, and hence any delays in this work would cause a ripple effect all down the schedule. However, in a clear case of Murphy's Law, the chances of there being a delay from the designer end was high (not always, but in most cases, there would be something pending).
How do you ensure that the work done by the designer was on time ? Well, there are no clear ways (atleast nothing that was 100% successful), but here are some steps:
- Layout a clear schedule for when the delivery from the designer is expected, including dates for interim deliveries, review times, and final deliveries.
- If the designer does not agree with the dates that you would have started out with, and considers them too aggressive, then you need to do a discussion. Don't try to force any dates on the designer from your end, make sure that dates are negotiated.
- Setup a weekly telecon with the designer, and make sure that you are reminding them of the dates that are due, and also find out from them whether they are on track or not. If wildly out of track, then you might need to modify your dates to some degree, and make sure that the team knows about.
- If there is a manager from the designer end who is looking at this, then make sure that they are also in the loop for the work done.
- Finally cross your fingers and hope that this is not the schedule where the designer is going to delay their deliveries.

Dealing with dependencies that can impact your project schedule ..

Dependencies are the bane of a project schedule, or as one of the developers in a team told me, dependencies and other risks are the reason that a team needs a Project or a Program Manager. I have had informal discussions with some of the senior members of the team, as well as the members of the management team, and there is a lot of discomfort in handling the various dependencies that bedevil a project; as well as doing all the negotiations that are required to mitigate the risk of a dependency, or when the dependency causes a problem to the project.
What am I talking about ? Well, consider the fact that in today's world, most projects have a lot of dependency on items outside the control of the core team. You may be getting some work done from outside the core team by an external team, you may be picking up a component that has been created by an external team (which can be somebody who you pay for delivering the component, or you may be picking up a component that is being done by a open source type of team which produces a component and you pick up the component as and when it is prepared).
However, anytime you are using an external component, you are exposing yourself to a dependency and to a high amount of risk (actually the risks depends to the extent on which the dependency can screw your project). Let me list down a few examples:
- You use a component that is fairly stable (say, being done from another team in your organization). The component is fairly stable, and even though new versions of the component are released from time to time, there is no major necessity of picking up a new version of the component. The dependency is there, but the risk is low and so the mitigation required is also low. Even if there is a problem with the latest version of the component, you can stay with the version that you already have.
- You use a component where there is a bug fix that is needed for solving a problem in your application. In such a case, the risk to your product or project schedule depends on the critical nature of the defect in your application. If the defect is critical to your application, then you have a high amount of dependency and need some mitigation plans. If the component has some problem, then you have to consider the impact on your project and have mitigation plans accordingly. If you are able to survive with the defect in your software, then you would not imperil your schedule if there is any problem in the delivery or the quality of the component.
- Now consider the case of a component that is critical to your product, or delivers a functionality that is critical to your component. For example, when you consider the Adobe Camera Raw and the products that incorporate it, one of the critical advantages that a new version of the Camera Raw provides is support for new Digital SLR's that have been released. So if you were releasing a new version of Photoshop or a new version of Lightroom, it is important for them to integrate the latest version of Adobe Camera Raw because of this support for new cameras. In such cases, if a component is delayed, it causes significant problems for the product schedule and needs a high degree of mitigation. In this case, the mitigation that is done by these products is by enabling this component to be updated later. However, the mitigation strategy for every such component or dependency may not be so easily solved, and the team needs to ensure that there is a lot of thought to how to mitigate these problems.

Communication: Informing people when there is a problem, and when they depend on you ..

You are providing a service to some clients, and they have come to totally depend on you. Like the post before this one, if you are providing analytics information to external teams, you have a lot of responsibility on your head. They would come to depend on you and if you are doing a good job for them, more and more, they will not think about any problems at your end and will depend on you to inform them if you are running into any problem. And that places an awful lot of responsibility on your end. Towards this purpose, you have to have a strategy to figure out what to do if something goes wrong at your end, and informing your clients quickly should be a part of such a strategy,
We recently had a big shock when it came to our analytics program, where we were using an external component. We were integrating such a component, and passing it parameters from within our code. What this ended up doing was passing data from our user base to the database maintained by the external team, and they would let us mine this information through a web front end, and they would also help us if we needed some new queries or new reports.
All this was perfect, and we did not bother anymore to check whether there could be a problem of any kind (so part of the problem was at our end before of a lack of checking for variables that could impact the program). We were looking at some data pursuant to the launch of a new version of the application, and in the second to fourth week of the launch, we saw data that indicated that the number of people using the older version of the program had reduced. This enthused the Product Management and other senior stakeholders, and a decision was made to target the older users through an email campaign sent to those users who were on our email list and had purchased previous versions of the program. This campaign would build on this conversion and offer some benefits if they did an upgrade, the idea being that if there are people doing an upgrade, it would help to push even more people.
This campaign was made ready in record time, and pushed live for a week. Like any other campaign, this program had some large costs because of the resources used, and the infrastructure cost for tracking of the email campaign, including measuring the conversion numbers. However, at the end of the week, it was very surprising that the conversion rate did not show any uptick; and in a coincidence, a day later we received an email from the analytics team that they were suspecting some configuration issues on their side, and they have probed, and finally figured out that some data had been lost, which also overlapped the weeks that made us decide that people using the older versions had dropped.
This was like a big blow, and since the analytics team was part of the same organization, we bit back the strong words, put in the issue for being more careful the next time that we did a campaign based on data. However, senior stakeholders did not hold back and had an interesting discussion with the head of the analytics team about not sending out a message to the various client teams when they suspected some data issues. The analytics team wanted to be sure before sending out a message to the client teams, but it was a learning to all of us. Since none of us really knows when another team is in a critical position and are dependent on us, as soon as we run into some problem, it is imperative to quantify the risk position and send it out the client teams that are dependent on us. This helps them avoid business decisions that are impacted if the data turns out to be problematic; even if it was a false alarm, there is some loss, but in general, the loss might be less than making a decision based on data that is incorrect.

What are the causes for the failure of traditional planning approach?

The traditional planning approaches does not always lead to very satisfactory results.

Causes for Planning Failure

Cause #1: Planning is done by activity and not feature
- The traditional approaches to planning focus on activity completion rather than on delivery of features.
- Activity based plans generally lead to projects that overrun their schedules.
- Hence, quality is reduced.

Cause #2: Activities do not finish early
Cause #3: Lateness is passed down the schedule
- Traditional approaches being activity based, their main focus is to focus on dependencies between activities.
- Testing will start late if anything goes worse than planned according to traditional approach.
- Testing will start early if everything goes better than planned.

Ways to avoid late start of testing are:
1. User interface coding finishes late.
2. Middle tier coding takes longer than planned and finishes late.
3. Middle tier coding starts late as tables adding to database finishes late.
4. Tester is not available.

Cause #4: Activities are not independent
- Activities are independent if duration of one activity does not influence the duration of another activity.
- For independent activities, late finish on one activity can be offset by an early finish on another.


Cause #5: Delay caused by multitasking
- Multitasking exacts a horrible toll on productivity.
- It becomes an issue once a project starts to have some activities that finish late.
- Dependencies between activities become critical.
- For a traditionally planned project, multitasking becomes a problem for two reasons:
1. Work is assigned in advance and it is impossible to allocate work efficiently in advance.
2. It focuses on achieving high level of utilization of all individuals rather than on maintaining sufficient slack.

Cause #6: Features are not developed by priority
Cause #7: Ignoring Uncertainty
- We fail to acknowledge uncertainty in traditional approach.
- Ignore the uncertainty about product.
- Assuming initial requirement analysis will lead to complete specification of product.
- Ignoring uncertainty about how we will build the product.
- The best way to deal with uncertainty is to iterate.


After looking at the problems with traditional approaches to planning, many projects are disappointing. Planning based on activity diverts us from features and as a result, a variety of problems leads to the likelihood of delivering late against a schedule.

How a task network is defined for a software engineering project?

The task network is a useful mechanism for depicting inter task dependencies and determining the critical path. When more people are working on a software engineering project, tasks and sub-tasks have their own inter-dependencies and also as the number of people increases, the development activities and tasks are performed in parallel. These concurrent tasks must be coordinated so that so that they will be complete when later tasks require their work products.

What is a task network?
It is also known as activity network. It is a graphic representation of how the tasks are flowing in a software engineering project. There is a pattern and a sequence that is followed by tasks. This task network can be used as a mechanism or a way to know this sequence and pattern of the tasks that can act as an input to an automated project scheduling tool.

On a macroscopic view, task network depicts the software engineering tasks. On a microscopic view, each and every task in the task network is expanded. For a concept development project, task network includes the following tasks:
- Concept Scoping
- Concept Planning
- Technology and Risk Assessment (It constitutes three tasks).
- Proof of Concept
- Concept Implementation (It constitutes three tasks).
- Integrate the three tasks which constitutes concept implementation.
- Customer Reaction

Software engineering activities are concurrent in nature. This nature leads to scheduling requirements. The project manager should keep in mind the following points to ensure continuous progress:
- He should determine the inter-task dependencies.
- He should be aware of tasks that lie on critical path which means the tasks that should be completed on schedule if project has to completed on schedule.

How to assess alternative architectural designs?

There are many architectural alternatives that need to be assessed. There are different ways to assess the design:

Trade off Analysis Method
It establishes an iterative evaluation process for software architectures. It involves six steps:
- All the scenarios are collected.
- Requirements, constraints and environment description are elicited.
- Architectural styles or patterns chosen to address scenarios and requirements are described.
- Quality attributes are evaluated.
- The sensitivity of quality attributes to architectural attributes are identified.
- Critique candidate architectures using sensitivity analysis.

Architectural Complexity
In architectural complexity, the dependencies between the components are assessed within the architecture. Dependencies can be divided into:
- Sharing dependencies represent dependence relationship among consumers or producers.
- Flow dependencies represent dependence relationships between producers and consumers of resources.
- Constrained dependencies represent constraints on the flow of control among set of activities.

Architectural Description Languages
Architectural description language provides a semantic and syntax for describing a software architecture. It is a set of tools and notation that allows the design to be represented in an unambiguous and understandable fashion.

Designing Class based Components - Component Level Design Guidelines

As component level design proceeds, there are some guidelines other than basic design guidelines. These guidelines are applied to components and their interfaces, their dependencies and inheritance.

COMPONENTS
- Naming conventions should be established for components.
- These are specified as part of architectural model and then refined and elaborated as part of component level design.
- Architectural component names should be drawn from the problem domain.
- It should have meaning to all stakeholders.

INTERFACES
- Provides information about communication and collaboration.
- Lollipop approach to represent interface should be used in combination with UML box and dashed arrow.
- Interface should flow from left side of the component box for consistency.
- Only relevant interfaces to component should be shown.

DEPENDENCIES AND INHERITANCE
- Model dependencies from left to right.
- Model inheritance from bottom to top.
- Component interdependencies should be represented via interfaces.